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ABSTRACT ^ . / ' 

Regional Interstate. Planning Project (RIPP) 
participants? ae'et periodically at bonf eren;ces Sponsored by ten Bpcky 
{fountain area state departments of education to discuss issues of 
general concerp. A aeeting was held July 21-23, 1976 .in Denver, 
Colorado, to consider various aeans of effecting the educational 
future through evaluktidn. Excerpts fih>M this seainar are presented. 
Oblectives for the conference included (1) the identification of ' 
probable directions of change in * education, and of vays in vhich 
evaluation could facilitate such change, (2) the shar^n^ of 
experiences and ideas on evaluation activities aao.ng the RIPP states,* 
and the disseaiqation df evaluation findings, (3) the review of 
pz:oBising or emerging evaluation practices, designs and aodels vhich 
^■ay be helpful to participants in their future work, ^and (4) the 
exchange of strategies which can be used to replay, v^ecycle and renew 
educational pr;ograas. Results of a questiLonnaire levaluating the 
s^.ainar are appended. (Author/BT) 
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FOREWORD 



The Regional Interstate Planning Project is a corVsortia of ten State Educa- 
tion Agencies that sponsor^ seminars on topics of^mutual concern ^o in-service 
their personnel along with local education agency staff. After the seminar 
topic Is §elect\ed, a Planning Conmittee, composed of knowledgeable staff from 
sevej^l of the SEAs, is selected to author the seminar objectives, plan the \ 
activities, select presentors, etc. ' j • ' . * 

This publication contains the proceedings o;^'a seminar on EvaluatiSh for 
Effecting Educational Futures. To expand on the topic the following objec- 
tives were written to be' achieved during the semin|^r: ^ 

• 

1. Participants will be able to identify probable directions of change 
^. in\education and ways in which evaluation should facilitate such ■ 

chartge and should assist in overcoming obstacles or roadblocks to 
> change. • . 

2. Participants will be able to share experiences and ideas on eval- 
uation activities among the RIPP States and to disseminate evalua- 
tion findings and reports. 

3. Participants wgll have an opportunity to review promising and/or 
emerging evaluation 'practices, designs, and models which may be 
helpful to them in their. future work. ^ - 

: - i ' ■ 

4. Participants will gain ideas and cojicl^ete suggestions on utiliza- 
tion strategies 'which will increase the chances that evaluation 
findings will be. used toVeplay, recycle, and renew educational - 
programs' 

In order' to achieve the objectives, the following individuals were engaged 
to make presentations to the seminar participants, 'to present their evalua- • 
tion models, with emphasis on the design, findings,' util Ization strateg/{es- 
and prediction regarding the outcomes: Dr. Kast Tallmadge, Vice President, : 
•RMC Research Corporation; Dr. Jim Vanecko, Senior Social Scientist and Project 
Director, Abt Associates, Inc.; Dr. Launor Carter, Project Director, Systems " 
Development Corporation; Dr. Roy Forbes, Director,. National Assessment of 
Educational Programs, Education Commission of the States.- 

William R. Bronsbn, State' Director, Office of Program Evaluation Research, 
California Department of Education, (flscus"Sed with the participants how the 
material for training ©valuators was developed and how It was used to do the* 
actual training. ' > ^ ■ ' / ' 

,Dr. Arthur Coladarci, Dean, Sfchoo.l of Education, Stanford University, gave 
ithe Keynote address that^stlfnulated the participants regarding the Importance 
lof the seminar topic* .• He challenged us to define "Education", and the basic 
jsubjfects to be taught or taken In obtaining an education." He also gave us 
{food for thought In . challenging us to be aware of where .an 'individual will 
Ipbtain the necessary Information io function as a productive, worthy person 
'in the twenty years and beybnd. . " > . . , 



Dr. Gerald Kowitz, from the University.of Oklahoma, was-.our concluding presen- 
ter. He .tied the, activities, which we'had conducted during the seminar, . 
together and* gave us advice on how to^plan in the future t6 ^improve our project 
activities, ^ - y . ^. / ' ' 

• ' ' • ' ^ . .) . 

In addition to the above, M would like to tfiank^Pat Watson, Okfahoma, Bea 
Smtth, Texa*, and Jim. Miller, New Mexico, vor th#r forthright statements during 
their reaction regarding the evaluation models. 10 the representatives' f^om 
each of the States who' made the presentations during the Pack and Brag sessions, 
' to the State Coordinators for theij^ assistance in planning, coordinating, 
moderating, taking pictyres and hejping in many ojther ways, your help and 
. assistance Is.jippreci^d, To Verl Snyder and Harry Phillips, thanks for • 
y. ^ ^the informative., 'helpful bViefihg regarding education legislation. To all, 
' # a big th^ink'you for a job well done, • , 

J • ■ . ■ • - , , 

Lamar ^LeFevre ^ ' 
^ Director • 

Regional Interstate Planning Project 
Nevada State Departmen-t^ of Education ^ 
f 40^5 South Spencer, Suite 234 
Las Vegas, 'Nevada' 89109 
(702) '385-0191 



. • „ INTRODUCTIPN - 

The ReglonaM^terstate Planning Project participants meet periodically at 
conferences sponsored by ten State departments of education to discuss new 
or topical Issues of'genjeral concern. The most recent meeting was hejd 
Ju.ly 21-23, 1976, at Denver, Colorado, to consider various mean^ of effect- 
ing educational futy>e through evaluation. 

The R.I. P. P. Conferences are financed with funds provided under the E-l«nentary 
and Secondary Education Act of 1965, Public Law 89-10, Title V, Section 505 
as amended. The project Is administered by the Nevada State Department of 
Education under the supervision of Superintendent John Gamble who serves as 
Chairman of the Policy Board. Other cooperating State departments of educa- 
tion are Arizona,; Colorado,* Idaho, Montana, T<ew Mexico, Oklahoma, Texas, 
Utah, and Wyoming. . " . 

This particular Issue was plagued with problems resulting from record'ing 
equipment failure during the conference. In most cases the speakers were 
gracious enough to provide us with' assistance In recapturing their presenta- 
tions. However, a few were pot able.to provide them in time for publication. 
The editors- apologize for this ^nd will make every effort to -avoid similar 
occurrences/ In- the future. 

During the process of transcribing, editing, and compiling the R.l.P.P. 
proceedings ,-the editors have sought to keep the material both brief and 
Informative. The.resultlng report is, hopefully, a useful record of many 
thoughtful discussions. 
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Dr. Jake Huber, Co-Editor 
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,Dr. Len Trout, Co- Editor * 
Research and Educational PlannVng Center 
College of -Education 
University of Nevada 
Reno , tievada , 



. • TABLE OF CONTENTS 

. , ' PAGE 

RIPP COHFERENCE PARTICIpj\NTS . ; 'w, 

PHOTOGRAPHS. k . vii 



SUMM/|RY--EVALUATION: SANCROSANCT AND PROFANE 

(Di'. Arthur Coladarai) 



t ' ASSESSING THE COGNITIVE ACHIEVEMENT BENEFITS RESULTING FROM ESEA 
TITLE I PROJECTS: THE PLAN FOR 1977 ANDBEYOND 

(Dr. Kast Tallmadge) • 

RESEABCH ON DEMONSTRATION COMPE/JSATORY EDUCATJOM; STUDYING 

IMPLEMENTATION AS A GUIDE (tO POLICY (D^. Jim Vaneoko) . . . 



CALIFORNIA EVALUATION AND IMPROVEMENT PROJECT 
(William Bronson) . . . 



SUSTAINING EFFECTS OF COMPENSATORY EDUCATION ON COGNITIVE GROWTH--A 
• NATIONAL' STUDY (Dr. Launor Carter) . 11 

NEWS FROM THE POTOMAC ^ • . ' ' 

(Harry 'Philiips) . . . . . i ' , 16 



A DISCREPANCY -BASED PANNING MODEL USED IN NEVADA • . ■ ^ 

(James \Kiley) . . '. . , , . '. 18 

ASSESSMENT MODEL FOR VOCATIONAL EDUCATION IN NEVADA 

(John'Madson) 2.'.^....; •' 20 

STATEWIDE EVALUATION OF INSTRUCTION^tti^ROGRAMS IN NEW MEXICO 

(Alan Morgan) , 22 

EVALUATION EFFORTS. IN THE STATE OF TEXAS-CAREER EDUCATION, DEVELOPING 
READING AND MATHEMATICS OBJECTIVES, EVALUATING COMPENSATORY 
EDUCATION (Dv. Walter Bouard and Carl Defibaugh) . . 25 

UTAH'S STATE ASSESSMENT^RROGRAM ' ^ . 

(David nelson). . . ' -n- '. 27 

I ^ • , . . 

COMPREHENSIVE-SYSTEMATIC PLANNING IN WYOMING 

f^homoB 0. Kroner) ^ : .' 29 

EVALUATION OF- CAREER* EDUCATION IN ARIZONA 

■ (Richard Ruff) . ' ! .■ . * 3r 

INVOLVING ELEMENTARV AND SECONDARY TEACHERS IN DATA UTILIZATlQf^ , 

(Bill Comett). ' : , -33 

EVALUATION ACTIVITIES OF THE PLANNING. RESEARCH, AND EVALUATION ' 

SECTION, OKLAHOMA DEPARTMENT OF EDUCATION '(Charlea Sandmamh . . . ^ 34 



ERIC ' - 



iv * 



'INFORMATION UTILIZATIQN IN DECISION-MAKING^ 



(Dr. Roy Forbes) 



SUMMARY, EVALUATION AND IMPLEMENTATION 

(Gerald Kowitz)., . J- 



EVALUATION 



RE6I0NAL .INTERSTATE PCANNiNG >(^)JECT * 
CQNFERENCE PARTICIPANTS 

Denver, tolorado 

July 21-23, 1976 "\ 



ARIZONA ' 

Jerry George ~ 
Don Kearns 
Richard Ruff 
Beverly Wheel pr 
David, WiUiams ' 



COLORADO 

Bill BroWn 
Dwi^ht Cool 
Cathy Felkpor. 
Roslyn Grady 
Ed Steinbrecher 
Bob Willard 



IDAHO 

Re id BtShop''- 
John Briggs 
Donald Carpenter 
Bert Stoneberg 
Helen Werner 



MONTANA 

Bill Connett. 
Ralph. G. Hay 
Don Moore 
Phil Ward 



NEVADA 

James Bean 
Richard Gunkel 
James Mlel^ . 
LaMar LeFevre 
John Madsop 
Courtney Riley 
Shirlee Wedow 



NEW MEXICO 
f 

Allen Ferrel 
Jfyce Goodwin 
James Miller'' • 
Alan MorgOn 
Roberta Stuart 



OtaAHOMA 

Gladys Dronberger 
Milton Ford 
Frank Hobbs 
Victor Kehn 
Gerald Kowltz 
B«)bby Martin 
Charles Sandmann 
Pat Watson 



TEXAS 

CBrI Deflbaugh 
Walter Howar*d 
K6n KyM 
Bill Marshaa.l 
Bea Smith ' • 



UTAH 

Richard Burbldge * 
Bill Cowan , 
'\ W111ia<n Hutchinson 
Nola Miller 
David Nelson ' 
Don Richards 



WYOMING , 

Mark Fox 
Tom Kraner 
Marcia Troxel 
Pa^ Wunnicke 

GUESTS 

B'111 Bronson 
Launor Carter 
ArthurvCbladarcl 
Betty Rellbert 
Joe Lavter 
Verl Snyder 
Kast Tallmadge 
Len Trout 
Jim Vanecko 
Roy Forbes 
"cJames Mortensen 




X * Verl Snyder, USOE Representative 
^ for the RIPP Project, discusses 
'Intersrtate Cooperation. 




Jim Kiley, Nevad^, makes a point 
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•Dr. Kast Tal\madge, RMC 
tion (back rightj makes 
interested participants 
benefits resulting from 
Projects- 



Research Corpora- 
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about achievement 
t'he Title I 
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•Dr, Mark Fox tells. the partici- 
pants how the Wyoming State 
Education. Agency uses evaluation 
data, to inform Legisl-ators. 




Dr.'' Laurvpr Carter, Systems jOevelop- 
ment Corporation, emphasizes a 
point regarding effects of- Compen- 
satory Educatiqn on Cognitive 
Growths 
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Verl Snyder and Dr. Harry- Pdi 11 jps from 
the USOE keep participants abreast of 
Education Legislation being cohs;i^dered 
irv Washiagton, DC. 



Dr. Beverly Wheeler, Arizona, out in, 
front of the Ari^na Delegatioa.' 




Montana Delegation - Phil Watd (Center) 
Montana Coordi-nator. 




\Dr«& Roy Forbes, -National Assessments of 
■duc^tional Programs, Education Commis- 
sions of the States, discusses how to 
uiilize data to make decisions. 




Montana shows Evaluation Models. 



\ 




Wyoming Delegation - Pat Wunn,icke, 
CpordinatcJr* (thiTd from. left) . 
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Colorado presents its study and operation A 
procedui;e for concept six-year around sch^ 
programs. ^ ' ^ . • — ' " ^ 










If 






/ ■ 







LaMar LeFevre, RIPP 
Projeo^ Director, opens 



a session. 




Dr.. Ed Steinbr^cher, Arrangements Chairman Drl Jim Vaneckp\ Abt Associates, 
and RIPP Coordinator, stands on the right " chats with Pat Watson after 
.of the'Colorado Delegation. ^ presenta'tion 



X- 




Big John Madson (back row center) gets .Utah Dele'gation - Wi 1 tiam Hu*chi nson 



thp 'Nevada Delegation together 



(second from right) Stat-e' Coordinator 
of RIPP. ' 
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Idaho Delegatjon 
.Dr. Reed Bishop, (^^cond from 
left), 'Idaho Coordinator. 



Don Richards, Utah,i sjiares an 
Evaluation Mqdel with other ^ 
Stafe participants. 






Dr. .Walter Howard (standing), 
Superintendent Jim Mill^i 
Rat Watson, and Bea Smith react 
^ # ' to the Conference Major 
• Presentation. 



Four Major Presento.rs (left to- 
right): Gerald Kowitz, Verl 
Snydejr, Launo)^ Carter,- and . 
William R. Bronson; State Direc- 
tor, Office of Program Evaluation 
and Research,- Califoi^nia De^t. 
of Education. 

-• • 




Dr. 6ei*a1cl Kowitz, Conference Eyaluator (second 
from left), 61a(j)« Do*bergerj Memljer of the 
Planning Coninittee (third from left), and Pat 
• If/atson of the Oklahoma City Delegation. - 




i 



Oklahioma shows Evaluation Model's with other 
States. ' ' » I 



r 





Dr. Walter HowaVd, RIPP Coordinator from 
Texas, in the middle of the Texas Delegation. 
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SUMMARY ■ 

EVf\LUATION: SACROSANCT AND PROFANE 

\ ■ ■ ' 

f Dr. Arthur Coladard 
^ Dean, Scho&l of Education 

. Stanford -University 

The logics,' tactics, and techniques of "evaluation" are not Invariant across • 
situations. Rather they vary with the kinds of purposes held in view. At 
le^st these differing sets of purposes arise in education: 

(a) . Evaluation to make decisions about the adequacy of administra- 
tive, organizational strueturi and process. 

» " ^< 

^ tbj' Evaluation to make decisions about pupils (e.g., placement, 
'programming). 

(c) ^.Evaluation to make decisions about adequacy of the Instructional ^ 
* / programs (curriculum and teaching method). 

We often confuse the three, thinking of all as constituting the same evalua- 
tive task. 

Fociising on the third purpose, another distinction -should 'be made: (a) evalua 
tlon for the purpose of "testing" the program (summat1\^ evaluation), 
(b) evaluation for the purpose of Improving the program ^(formative evaluation) 
in the first (tufimative) , the. canons of "science" apply--that Is we hold 
"sacred" the principles of rellabllfty, objectivity, reproducibility,^ etc. 
However, It 1s^ counterproductive to extend this "scientism" rigidly to the 
second (format'ive) mode. Rather, in continuous evaluation for continuing 
prograrti Improvement, we should learn to tolerate the "less scientific" as 
long as the information 1s_ the best that can be obtaiXed under the program 
conditTons. 



•>■• 



ASSESSING THE COGNITIVE ^CHILVE 
. RESULTING FROM ESEA/TltLE. 
THE PLAN FOR 19^7 AND Bl 



lENT BENEFITS 
^PROJECTS: 
'OND 



by 



' D''- Kasf TalTtnadge; Vice President 
RfC Research Carporalion, Mountain View, California 

Il^Iinf f^^^^ posttest scores as a measi/re of post- 

what U s^S'^nn■'^;^•^°" ^° ^'^^'^ ^ ^^^-^5' you feel Measure; 

?nJli LJJ that^you re fefying to.teacm.*We try to" generate some scores on tl 
total test which we th/nk they wouTd have, gotten if they had not pa?ticipat^ 

^?n'.^f r''''- ^" H^^^'^ ^ ^^^'"^ concept"-all use S^est 

scores as a measure of pd^t-treatment performance. They differ in teVws^how 
they generate the no-treatment expectations. ^£nttsj»T row 



model 
control 
del Z/H 
modific4tfon- 
to- generate « 
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The three models are labeled A, B.^^nd C. Model A is a nomVreferen 
It probably looks like what most people are doing today. Model B is/'a 
group model, and in. most Title I project's it is probably illegal, 
what we call a special regressiqh modfel. - We've developed a sligh 
or. that whi-ch we call regression pnojection model. We use tha 
a -no-treatment expectation, over a particular set of circumstanc 

• In Model A we use published, normative data. We use it in th^same waV that wb 

?n thp"Jl° "^^P • ^ 35 they wela Wal case 

in the control group. Instead of comparing k1ds against "thf national norm" we 

Wo:S?::?^'''''''";' ag^tnst^kids ?n th,. normative K^w^ go The- 

same pretest score?. We generate thi|Jow treatlhent expectf- ^ 

simple way; We assess at plt^test time the "percentile stanc 

respect to the national norm. We then assujne', all other X\ 

they will maintain their status" with respect to their'grad 

special help over the pre-' and posttest time period. In oj 
: !J^''L?u* percentile and you don't help them, 

tne ZOth percentile. Many people' jump at this exp^ectatior 

our kids fall farther and 'farther behind the national norr 

They fall .farther and farther, behind the national norm be( 

same percentile. They falJ. fartfier behind in raw score pc 

farther behind in standard score poms; they fall fartheW 

equivalent sqpres* bt>tthey maintain iheir same percentile 

^^I! J* be aVly^TDther wij*.. What we are saying iL mat tney stay 

.soir^ewhere between the kids that ai^ just a littl«bit bfigLr than they are . ^ 

we ca-l L^J^'hl! *'];"1d^.b1t.dumbey than they are. ie assume that unlesT/ 

tP.t ni!^ them, they will maintain their pretest percentile status at post- 

nocJficr* ' get a,no-treatment.expectation from the expected 

posttest sc,ore.^That's our no-tr^eatment expectation. cahcv-lcu 

lrn^2c fH* *° use this -norm-referenc& modeU ypy have to do it With' standard • 
scores and percentiles and not with grade equivalents. You .have to use percen- 
p!;Snii J !!T ffP^r.^cally deriyed. In other words, percenliiles which were not • 
extrapolated by the test publishers. The California" Achievement Test, for ' ^ 
example, was normed in February. So the' percentile norms of^he California ' 
Achievement Test m February are empirical norms; The test pSbllshers present 



tion in/ a very' 
Ing of k1ds w.1th 
ings being equal , . 
■level peers without 
pher yords. If they 
phey will end up at 
saying, "Then why do 

This is true, 
iuse they st^iy at the- 
ints, they fall 
behind in grade- 
Nationwide they 
that they stay 
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in their manual^ all the spring norms in addition.* But those norms are con- 
structed by 'linear interpplation^ between Februaries, under the .^assumptioa that 
projectordwth /is linear over the nine months of the school year, with one- 
thir4i as much ^ain being made over the three summer months. That^assumption 
gives you a nice linear growth over the arttficial ten academic-mont^ yea*^, but 
it Is 'simply untenable. 

It's pa^-t'icuUrly untenable with disadvantaged and g^ted kids- because disad- 
vanta^^\idS[' decl ine in their growth* rate over the summer and gifted kids 
exceed their prowth rate. In fact, if yoi/'were to take data from any test 
(Publishers' norms^and plot the growth, i.e., get a student at a standard devia- 
tion belo*4;thfe mean (the 16th T^ejcentile) ^and one at -a standard deviation above 
the ipean n^)^,4th jrfercentTle) and an average student at'the 50th percentile 
and plot themover«e school year and then over the summer and then over the 
next school ye,ar» yqtfwill see thSt all three grow very much the same over the 
school year. The df^ereiKe bet\JSfeen them occurs during the summer. The dis; » 
advantaged kids ^5111 fartTier beh.incl,^t1i6»;5national norm kids tto what they are 
supposed to do, Snd the gifted 'kids do -a whole lot .better1:han the^ dc/ while 
they are' in school. • * ' \ . 

rm*not a^jwiming that the di.f.ferent kids in the norms are in different programs 
I'm just sly^ng that'you- have to use empirical norms that set some additional 
reqiyffements as to when you can test. If you want to use the California 
Achievement Test, it only .has empirical norms in February.- That'is the only 
time you ,can test your kids and make valid comparisons against the-norms. You 
can't test in the fall or spring because the norms are proje'cted and the errors 
in projection ara sufficiently large to invalidate the norm reference of 
cpmparison. ^ - 

We were able to cbnv.ince the California Achievement Test publishers that^their 
projected norms are, not adequate. They are systematfcal ly in error. The dis- 
advantaged kids were systematically too low in the fall and systematical ly too 
high io the spring, * The Cal ifofni a ^Achievement Test is cOrrentty being 
renormed. We take'some' modest credit for that. The CTBS 'falls in the sarpe 
category. ^ For educators who want to use *a fall ^pretesting and spring posttest- 
ing model at the present time^nly two standardized achievement tests can be 
used in conjunction with the^inphfn-reference .model . 

One additional requirement that we make in conjunction with Model A, our norm- 
reference model, is that you cannot use a single set, of test scores to Select 
the 'kids who ar^ going to be in the treatment and to pretest. If you use the 
same measures to select kids because they have. low Scores and then use those 
scores for pretext measures, you have random error-working in your favor. You 
can get statistical regression toward the mean which adds up to whatever treat- 
ment effects you might have .had, and you get too big a mean. We requii^e that 
you select the kids fitfst by whatever method you want." Once you have seVetted 



Model C is quite complelc. The positions of each student are plotted on two 
axes, one axis for pretest scores and one for posttest scores.. Whenever we 
pretest and posttest a group of 'kids 'and then plot their positions in^this^^ 
manner, we will get an array that looks like a scatter plot, It iS'^so charac- 
terized because it is elliptical in s^hape and the ca^s tend to cluster on the 
sevfen. If. we look at any particular test'score and make a slice that corre- 
sponds to a pretest score, and we look at the midpoint of that slice, we should 



them, then you must pret^pt them. 




have exactly half the cases from that score above the IW^and 'half below. 
If we look at the distribution, of scores, the distribution of posttest scores, 
; and the. kids- that got this pretest score, weJshould have something that approx- 
'jmates normal distribution. The mean of that distribution is the midpoint of 
•■the slice. Any place we make a slice, we should secure the same results. - Or, 
.;if w6 draw a line between the midpoints of all the .sUces, we secure a regrea- 
$ion line. Basically the regress.io(i line is theUnfe that goes, through the 
fftidpoirvt line of all the slices, and. in all cases it-gives us the best pre- 
diction of the posttest score at a pretest score position. The symitietry at 
this array is what makes the special regression model work. . / ' 

• . ■ r ' ''■ . 

With this model we can predict posttest scores- for hypothetical kids by knowing 
pretest scores. If they have a certain pretest score,' then they/ should get a 
predictable posttest score. Suppose we administer the pretest /to -1,000 , kids, • 
but we only have enough money, to serve 300 of them in a specia/ program.' We 
want to obey^he law and, serve the neediest ones;, so we're go/ng to try to get 
the lowest pretest score uHtil we'get 300 kids. ^ There we wi/l draw the line. 
Only kids below tbat ^ine get the Title I treatment.- None df the kids above 
get' the treatment'. ' " 



Modejs B and C both r.equTre testing non-participants. This is the main objec- 
tioo to them. Another ebjection to Model B is'that you 6an' t really find 
kids comj^able to Title -I Icids'unless they 'are in a si^igle semester-long 
..program. ^'TWodel C is also, fair>y complex in tems of caflculations , and some 
people ha^'e objected to it for that .reason. So/ie people object to Model A 
because of -the restrictions on testing' time— d/uble tfestfng in Order to select 
any protest kids. And there -is some stigma a^ached/ to using^ational norms.- 

The .Educational Amendments of 1974 s-tate th^ the Commissioner shal 1 "provide 
such technical and other assistance as may /be necessary to the Stat*»education-' 
al agencies to'en^ble them to assis't. locaV educational agencies in the .applica- 
tion for artfi the develop^ient and systemat/c evaluation of programs in accor- 
dance With the models developed by the ccimmission. . This assistance is going to 
take two forms during this coming year./ First there will be a series of 11 
three-day worksrhops', one in Washington/ D.' C. , for the, "feds," and one in each 
of the regions,, primarily for State-leVel ev^luatop peoplfe, . State Title I 
people,- and State Title I, evcluators./ Each of these workshops is intend^ to 
train -people to become conversant with the system and also to prepare them to 
conduct similar workshops for the L/A's within their States. 

The second form of -technical assistance will be technical assistance centers— 
again, one for each of the region^. These centers are-to provide essentially • 
free consultant service? to theStates and the locaT agencies within each ^ 
re.gion. The States will call uphn these agenciesrto conduct workshops, to help 
them with wQjjkihops, and to helb with placing and proposals to those centers. 
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The research Lam here to dBscn^^e yoU, i^ a study irt distributional equity. 
It^is a stu^y which'departs frofl^^lE^aditiohal, educational Vesearch but which/ 
could not be mora: in the ^Center of thfe research tradition. . It is a study' f * 
derived* ffqrn^ij^x+ijmo thefts that Tf pompensatory ed^frerftionaf services are 
distribu»€wl''<m\tiie,Msi^^ of educational disac|Vantage (achievement test, scores)*; ^ 
rather thari on 'tf|5>basis of fro^verty, then, more children from poor families 
will , receive compensatory services and poV children will receive more services. 
PerlJaps the hypotAbsis 15 facetious, -perhaps merely provactive, but it is ^ 
interesting and t^stabler - . ' . 

If one cpasiders that current regulations playjiavoc in desegregated -^dhpol 

districts so 'that money is allocated to schfiols biegause of th6 poor kips 

who live nearby but go to 'school elsewhere^, ijF. one considers' that a poor crtild v 

who can'*t read tut lives on the-east side' df .the stre^, may be denied services , 

becfiusp^ many^hildren who are also' poor btit can read live on the we§t side 

of the sjtreeu^hen it is not an unreasonable hypothesis.' We social scientists 

ca]l it' the eco^log-ical faltacy. . . . * ' 

Educational eqUity has been mu^ discujssed in recent years. Traditional research 
on the issue, can be characterized as following two ipfiroaches, the accountingt, 
approach. and the Coleman approach., ^^oth suffer from exactly the same problem— 
they do.n*t deal ^ith teaching^ with process!, jwitl^ implejnentation, with service 
delivery, withowhat actually gets, to kids<^ They don't deal with the organiza- 
tional constraints or the deVivery of equal education^il opportunity. The ' * 
accounting approach def ine^ .equT*ty in terms/of inputs— the distributibn of goods 
and services.. <^The Coleman apprpacf^ (.named after James -Coleman^) defines .equity 
in terms of outputs--the achievement" test outcomes. of education. Both in 
their worst forms a r6 black box^proaChes . The fact is that the concerp of ' 
educators^ and educational researcners aliike is with the educational process.^ 
The proceY^ requires M'itputs of resources, a^nd it must be judged in termls of 
outcomes; \)ut to fqcus on either or to assuW one .simply produces 'the ottier 
• is a mistake".- We have talcen care ^0 ^void that miitake. Only time. will tell ' ' 
if we have succeeded; ' / . • ' ^ . 

Background and Objectives ^ ' , . 

Title I has/eceived much criticism, ^particularly in the^reas of program 
adminlstrati'on, student selection, arid service, delivery. In an attempt^ to 
ad^ess thfese, tongress authorized the f^tiorial Institute of Education '(KIE) 
to conduct a major study of compensatory educ^ti^. NIE drafted a- plan tor v 
a comprehensive set research projects. The Study of Demonstration Title I^ 
Compensatory Education Projects, one of this set of Research projects, deals 



W3t;h.the effects of using educational need rather than economic need as the 
basis for distributing^ Title^ services. Sixteen Local Education Agencies ' 
contracted with NIE during the J975-76' school, year to devise plans for 
implementing Title I programs based an ah' educa*Tcrnal definition of need; 
Thbse plails^ accepted by NIE will be Implemented during, the 197^-77 and 197^7-78 
school years. , . • ' 

Theresearch portion of the Demonstration .Projects U being, conducted by Abt > 
Associates Inc. (AAI). This research, will compare^the distribution of instruc- 
tional an-d other educational services during, the 1975-1975 school year, under 
the current federal regulations, with the distribution during the 1976-1977 . 
and 1977-1978 school years, with Vederal regulations waived. It will provide 
the basic analysis for possible modifications \r) the Elementary, and Secondary 
Education Act (ESEA) of 1965 w'ith regard to allocation policy: eligibility 
criteria policy and distributive policy. ' . 

The definition of the study within^he set of studies planned by NIE, and the * 
districts' Demonstration plans have led to the establishment of five major 
research objectives. -They are presented * below in order of priority. ' ; 

1. To^estifnate and ajialyze the distribution of services received by. 

" . students. To achieve this- objective a log' "is completed by teachers 
-^..>f5r a sample of students for entire sample days. ^This log will pro- 
vide data on the f.ull rang^ of instructional services received by 
different type^ *of students: ^Ti^le L versus non^Title I students., 
. students from varying socio-economic and ethnic backgrounds, and 
students of varying achievement levels. ^ ' 

2. To estimate arid analyze the distribution of services delivered 

- across types of schools^ and types of students. To achieve th\s 
objective questionnaires ane completed by principals and teachers. 
These questionnaires focus on changes in programmatic content and 
strategies, as well a^ distributional* changes. f 

• 3. To estimate and analyzl oommunity reaction to the Titled programs 
and the Demonstration. To achieve this objective" interviews 'are 
conducted with parents of Title I children and members of P^rert^ 
Advisory Councils. 

4. To estimate and analyze th'e costs or savings associated wjth the 
. altered allocation and eMgibility,policies. ^ 

5. To ga-ther descrip'tive information on the possible achievement 
outcomes of the Demonstrations. • ' ' , 

Given the research questions ehumerated above and the preliminary district 
plans for the Demonstration, the sample^ was designed to include four types 
of schools: - • ^ * ^ ' 

1. Schools which received fitle I during 1975*1976 ^nd which. were expected 
to receive Titl.e I darin 
(1976-1977 and 1977-1978 



to^receive ^itl.ej^darin^ the implementation years of the Demonstration 



2. Schools which did not receive Title I during 1975-1976 but which were 

expected to receive it during the implementation years of the Demonstration. 
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3. Schools which, didmot receive Title I 1975-1976 and which were not 
expected to receive if during the ceurse or the Demonstration. 

> 4.-' Schools whicTi received Title I during 1975-1976 but which Were not ^ 
• expected to receive It during ttie "implementation years of the 
Demonstration. ' • - 

.^ce schools h3(ve"^been selected, all third and" foXirth grade claasr^oomS^ within 
these 'sch6a].s are included in the sample. Two students (if possible, one! 
Title 1 and one non-Title I student) are ip^ndomly selected'from these class- 
roomi along with two altfrnates for each primary student. Alternates are 
involved in the research only if the primary student moves, ts^sick during 
the-'data col lection period, or is unavailable" for some other reason.. 

Because the* research calls, for information on, the regular and, if appropriate. 
Title I Mnstruction in r^^dtng and mathematics delivered to these 'students , ' 
teachers who provide this instruction ar*e included' in the sample. Because , 
the research calU for information- on the brgt^ization pf the districts' ^ 
schools, the services they provide, ^ and the students they serve, all princi- 
pals In the districts are asked to participate in the study. Finally, since 
the research requires information about parent involvement in and reaction 
to. the Demonstration, 'Selected parents and Title I Parent Advisory Council 
members are asked ^to participate in tHfc/esearch. 

^ Basic Analytic Model ^ ^ • ^ ^ ' - 

• . • ' ' t ■ ' ' 

The basic analytic jnodel under>ying the Title 'Demonstration Stiidy coi).sists 
of groups pf factors outlined in the^objectives earlier^^nd the linJcs between 
tfiem that will' be analyzed in this research.- * 

The process leading from eligibility and distribution policy decisions to.^ 
program changes and, ultimately, to chang6sMn thetstuderits served as segffien.ts 
of.ihe model. r » , , ^ ' * 

We^hope that at the end jaf this study, we can say wJiat gets to kids as a 
resu,lt of different a-1 location policies.- The policies are the choice of the 
districts participating. What gets to kixis'is for us to ^determine, by focusing 
on implementation and the pNrocess of sSrvl^ delivery. .If these hopes are 
realized, wewill be ready-to say something about what, is reasonable policy 
and, more. importantly, what policy alternatives affect'kids. 



^ ' CALIFORNIA EVALUATION AND ' ' 
IMPROVEHENt PROJECT 

■ - , • / - . ■• • ■ ' : . 

• * ' - by. . ^ 

Wm.lam BronsDR, State Director , , 

O^^ice-pf Program Eva1uat,1oa Research 
^ California Departmertt of Education ■ 

•■ • One of 4 basic assumptions is that evaluation is too important to leave to ' ' 
the .lpca\^^luator. We have tp get data from the village watchman because • 
that IS wn?re)we usually get it--the problem is that the village watchman just 
puts down what he damned .w.ell pleases. What we need to do fs trai-n the watch-' 
man. ■ ' ^' ^ « ^ 

' " . * 

' ■ Icti tures we have in our proj,ect--becaus€L.the charge from 

ESEA Title IV. Part B, was tP ra^ise the skill at the Ipcal level In the area 
pf-prpgram planning apd evaluatipn disseminatlpn. Until prpgram evaluation 
practices are accepted and itnplemehted from the classrppm level tp the schppl 
Sati ^^'"^^ ""^^"^^ P.^°9^^'"5 "i^^e as a. result pf 

If any pfypu . read the Rand repprt pn hpw decisipns are made, I think ypu will " 
• . nLTZi Vlw-^'t^^'^ abput gping into evaluatipn fpr decisipn-making parppses. , 
_^ . Hpwever, I think pepple need tP learn hPW tP make decislpns based uppn 6valua-. 
^^TJlf.llV ^"^t^'^^^Ps one pf the reaspns that they have npt been using it 
?L? 1^ tl shpuld have is tha^it has npt been presented in a fprm 

• to thp'nTnn.l l^'*^''^ "'^^^L. f^^l"^ Hiust be Seen as something beneficial 
to the prpgram and npt as- an addltlpnal task dpne fpr spmep»«i else. Evalua- 

• dl?i. nnc Jn^JS '^^O'^^^ipn anfaping assessments fbr th^^purppse- pf making 
decis pns--fpr the purppse pf dpi ng something with the infpnnatipn. Assess- 
ment Is merely Ippking, measuring, and seetnjg exactly what spmethina -is We 
nave a St^te assessment prpgram, but it is npt an evaluatipn prpgram. We are 
. trying tP teTl the Ipcal districts It Isn't enpugh. l^hat ypu neld tp dp is 

develpp fpr ypurselves an evalyatipn prpgram which will give ypu the Infprma- . 
tipn ypu^ed In order tp make decislpns. In prder tp imprpve ypur prpgram 
We miist^pve evaluatipn frpm the realm pf mystery and Intrigue, take It put 
pt hig+ier statistics, and present it as a. useful tppl fpi-^ Icy-makers -and 
aecisiPn-fnakers. i . 

llLli^l 5' ^^^^ei^entiate here between evaluatipn Infprmatlpn and experimental 
research design. Evaluatipn infprmatlpn is secured frpm whatever pr whbever 
can be helpful tp me as I make a dec1s1pn-the classrppm teacher, the principal, 
ro^;.. ^ookm for ultimate prppf. I am npt lopking fpr an experimental- 

,n J ^"S^- T^T information Is ^oing to be helpful *tp me, 

inat s all I need. If I'm gping tP make some Pther, widef generalizations- ^fen 

Ji7ifl° TJ' '''' ^ ab°"^ teml of ^S aTf ity! 

?o Jli^;/"^ f^' i"^-'^ "^^"^ P"^P°^^ q'^^te different. We need . 
to identify very cjearly the purpose for \vhich we are doing an evaluation. ' 

rLS^rrS^H are going t*» be criticized on the bas!s of exper,1merftal 

research des^^n when perhaps that is not what we were ^tending. I was--very happy 

o . ' . -i 22 ■ ■ ' 
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to^hearlDr. CoJadarcfi comment that unreliable information was not immoral. 

Unreliable tests are acceptable when they offer information that is usefiil, 
OList because a test does not come up to the goals or reli)?bility standards, 
it still may be very helpful in finding out specific things . -'The use need^ 
to be iden1;ified and accepted for what it is and not extrapolated into some- ' 
thing it* is not. I direct my remarks here really at progranv managers and 

. t?eachers. ^ , \ - I 

Long-rangA^ interests are going to be be'st servelj when professional evaluaiors 
'are able^ secure appropriate, timely, accurate inf6nna.tion that the policy 
and decision-makers can understand. and trust. But many evaluatio\i reports 
are automatically suspect when they come out of State departments. \ Until ^e 
cah communicate in such a manner that can be understood and believed, then we 
0 might as well be doing something else. And in order to secure this time-ly * 
and accurate information, we need to go back to the village watchman level, 
/to the local Vevel . 

Gathering information at the local level is very difficti4t# Perhafps the ' 
' professional* evaluator plays a role as an interpreter between the classroom 
teacher and th& decision-maker. In a sense, he needs to be bilingual. He 
needs to talk the language of the teacher— the level at which the information 
is a)llected--and he needs^ to talk the langijage of the decision-maker. The 
person who is going to wr^ite the reports for -the decision-maker 'from the infor 
maticTn gathered in the classroom is going to have to understand each of these 
languages sufficiently ji/ell to pick'yp nie/e ounpes of dif^'erence. 

The federal evaluatjor is very important in this role, and we-are trying to 
enhance his role as we conduct our workshops. As you well know, it* is not 
going to be pt^ssible to develop successful evaTuators in a short time. Much 
training must go into it. We offer a three-day workshop\ but if you can turn 
out a statistician or an evaluatof in three days, I WouJd-1 ike' to know about 
'it. We have not been able to do that so far 'in-Cal ifornia. What we think we 
are doing is taking program managers, planners^ and administrators who have 
skills in these areas, and we're trying to sensitize them to the problems of 
evaluation so that when they see them, in their process of planning curriculum, 
they, recognize them as problems of evaluators. They are nt)t scared away from 
them, and they at least know how to handle them-or when to ask for help. That 
really is the essence of our training program. 

The structure of our workshop is such that we^are dealing p/imarily with small 
districts, the larger districts are just beginning to come to the workshops. 
When ESEA, Title IV, was first inaugurated, its objective wajs. to raise skills 
at the local level in the area of pl-anning^^ evaluation, 'and 'dis|eminatiOD. 
We began with the assumption that the money would dry up eventual ly*, When 
it dried up, we wanted to'^leave a residue of expertise around the State that 
would still be available to help. . * • - 

We went to our county offices of education first. They have been in business 
for some time and will probably be in existence long after the funding of our 
project. If we could train these people-to continue our, work, t,hey would then 
be able toworkjwith the district and cOuld train another cadre of people to 
; work at the level of th6 local districts. We identified eight colinty offices 
an(i designated two of these as development centers--the other six became 
sal^lite centers. The primary role of the development ^enters was to develop 
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the,.actual workshoi^- itself . The primary emphasis of the satellite center was 
on the dissemination, the actual conducting of workshops. The' State's role was 
coordination. / ' 

L6t tne teliryou just a little bit about the development centers and the people 
wbo lyere eniployed as directors. We said to the county superintendents, "Get 
us peopl/ who know their constituents, who can talk to ~the people and arV 
respec^d., ^e r^ly don't care at)out their evaluatipn skills\" As a result 
we got people in .there who were oriented toward administration, towards curricu- 
lum, toward planning— mostly curriculum and planning— and this was the group 
that developed tKe material . They decided what they needed; then they w^t 
,to the professional evaluators and said, "Hey, how does this sound? Does this 
/framewark contain .anything that could possibly be subsumed under the total area 
/ of evaluation? Consequently, each of .these centers is a separate unit. Some 
. were, developed in the north arid some ir the south. Although our format is not. 
completely identical, we have set up a coordinated' planning' system. Each unU 
IS an entity in itself; however, each needs to' be a part of the whole. | 

^You can'enter our total plan at any point and move'in^any direction to cover \ \ 
the entire^ context. If a person comes i it to you/and says, "Hey, I. want ar test \ • 
■ for'thi^/ you don't say, 'Tm sorry, he can't gi.ve you a test until you go > 1 
back there because that's where our project- start^." We have to be able to 
respond to the\need and say, "Testing is only part of the picture. Here^ 
is the rest opthe picture.. Here is what you're as.king for." Then we move in 
those direcyjons and 'show them there is more to evaluation than testing, more 
to evaluation- than writing reports, more than statistical analysis, more than 
needs assessment.. It is the. sum total of all .of these things. Until you get 
all, of these elements, you are not going to have an evaluation program. Here 
IS where you are. Now I. am going to help yoii with this.- - \ 

In our workshops we started out with three basic steps: plan, conduct, and use. 
- Wejl, that is simple enough. I think that all people can see that .you have to 
pVan ft, you^^have to conduct it, and you should be able to use it. What does 
the plan consist of? Well, you have to determine the evaluation purpose and 
requirements: Why are you doing- this? If you understand why you are doing 
It, it is much more clear whert you have to do. - If you are evaluating merely 
to fulfil; federal "State requirements, it's very simple what you have to 
do. If you are evaluating for public relations purposes,' that'restricts- what 
you need te do. If you are evaluating to improve programs, you have a different 

TOCUS. 

These kinds of determinatio'ns are very imfiortant as you look at our program 
planning. You can,'t really separate program planning and program evaluation. 
When you look at it in this light, should we use this kind of a format for sec- 
♦ ondary education reforms^ The planning requirements-are there, and they are 
there quite -specif ical-ly. Once you find out what you are doing it for, then 
you determine an evaluation p-lan and procedures. You have to obtain your 
assessment tools, you have to log the data, you have to- analyze the data, you 
have to report results, and then hopefully you Should apply the findings be- 
cause If there .is no application of Ihe findings, then you'shbuldn' t have . 
gotten that far in the first place. 
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• SUSTAINING EFFECTS OF COMPENSATORY EDUCATION 

ON COGNITIVE GROWTH-A NATIONAL STUDY t 

' ^ , / -by 

"Dr. iaunor Carter, Project Director 
Sxstem Development Corporation, Santa Monica, California 

Congress' asiked the, Office of Education some-time ago to fijid out . the answers to 
a number of\ijMe»tions plaguing them with regard to Title L Congress^ said, in 
effect,' "Hev, look. We've beerv appropriating a "billion and one-half to two' 
billion dollars a* year for Title ly and* the evaluation studies so far tend to 
be /somewhat Idisconperting/ There just isn't a great deal of ev'ldence that the 
Ti'We I programs have been terribly effective in^pi eking up the disadvantaged 
children." Bo' Congress ^asked^O.E. to do a defending study to tell them how 
effective Tiple I really is. . Secondly, Congress said,\"We want to |cnow whether, 
TitTe I funds are being properly distributed—whether they should be based upon 
economic disadvantagem^n t or wh ether they should be based upon educational 
disadvaiatagementl Il^^^s t fg a' t e " ttratTwcol) 1 e m for .us also." So these are two of 
'the major thingS^^^rat System Development Corporation is attempting to do. 
•p^tween ourselves and the-Office of Education we haye tried to formulate a 
number of policy questions whi£h we hope will be answered by this particular 
study. 

. These are some of the^different policy questions we are trying to^answer. 
First are some descriptive questions. For the country as a whole, ,what is the 
nature of compensatory education? What are the things that are being done in 
classrooms that are called "c'ompensatary"? Second, who 'receives what kind of 
compensatory education? We know what the regulations say about who should 
receive it, but who, in fact, receives it?. What -kind pf childr&n are they? 
What are their educational needs? What kind of economic di sadvantagement do 
they have? What Kind of racial or ethnic composition do they have? Questions 
of that nature. ' ' * 

Third, how can one improve the effectiveness of compensatory education? We ^re 
going to be studying many different kinds of compensatory education, and hope- 
• fully, at the end of the study wie will be .able to say these kinds of programs 
seem to be more effective than other^kinds of. programs. Fourth, what effect 
does the receipt of compensatory education have on edifcational growth? This is 
a question of-- if the child starts out at grade one at a disadvantage, what is 
the growth frbm grade one to grade twa, to grade three, ''or to grade four, and 
so on? Is there a quick catching up that^is held, or does it falToff? How 
does that relate to the different kinds of students that aVe receiving the 
programs-? , ' * ' 

Fifth,^what is the optimum duration of compensatory programs? Some people 
arguer that'you ought to have your compensatory* programs quite early in 
'first or second grade; others say compensation ought to occur later. Where 
should you compensate if you have a limited amount of money? Sixt+v, What 
happens when ttudent programs ai^e discontinued? If a student happens to be in 
a compensatory program under Title I and then advances to a certain T'evel of 
achievement, he ntey be ineligible for the program. What happens when he no 
longer gets the program? Does tie fall back down again, or not? 
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Seventh, at what grade level do conipen sato ry progrLs seem to be' most effec- 
tive? Eighth, how do home and local circumstances^ affect the child? Do the 
intellectpalv school-oriented characteristics of ^he home, seem to have an 
efffect.on the child and how? Finally, whaf seems /to *e- the influence. of summer 
school on the achievement growth of the ^hild. T^r some children summer school 
or the lack of summer school se.rves to be ? depressant; for others it seems to 
PICK them up. So, tho^e are the policy questions that hopefully, at the end 
of this studXj,jjie„'„U„Jb)^Aj3le«tcus^SGmething quite definitive about. As a 
result of what we are able to say, we believe that Congress will change the, \ 
Title Ilaws. From a broad educational poiat of view, then, this could be a 
very important study. ' i 

NOW, what is the overall design of this study? We sha>l address these nine 
policy issues that I have'just described and , we shall definfthe kind of popu- 
lation in scliools we are going to investigate. The schools will consist of 
three or four different kinds. .Our first study will be longitudinal; that Is,' 
we shall follow over- several years the students in 251 schools, but'not the 
schools. ^nd not the classrooms. These 351 "schools -have been selected in such a' 
way that we are quite confident they ar| a truly representative sample of all 
the schools in the United States. We are going to follow all of the students 
in the school, rrot just the Titje I students, not State program students, but 
all the students, whether they are disadvantaged or not, because we want to-get 
a comparison between the disadvantaged students and the other students. We are 
.taking grades one through six, but we are going to try to follow the students 
as they l^ve the sixth grade and go into seventh, eighth, or- whatever, to See 
what-hapTpans to them as they jeave the elementary grades and perhaps +he 
Title-'T^if'iDgram. - • ' • 

In addition, there is a group of 30 schools where--as far as the records tell 
us at l|ast--there >s a high level of disadvantagement." These schools are in 
poverty districts., but they have no compensatory programs or very low compensa- 
tory programs. You might think under Title I regulations such a circumstance 
would be impossible, but it appears that, indeed, there are such schools. So * 
we have identified 30 of them and these will also be a part of the longitudinal' 
study. 

The sampling plan involves a selection of these schools 'in terms*of geography 
(they are distributed throughout th^ country) and in terms of the size of -the 
LEA fronwwhich the school *comes . We really wanted to get at school size, but 
it turns out there is no national record df strttool size. There is no place you 
can go and say, "Hey, what is the size of Emerson School?" You can go and say, 
"What is the size of this LEA?" So we had to get LEA sfee to substitute for 
schooj size, and then develop a pov^y index, We now have all 64,000 schools 
in the country categorized by 'the level of poverty of their LEA. 

The instruments we are usirfg, for students have to do with measures of -aca'demic 
achievement, measures of functional literacy, measures of attitude and self- 
concepts. We are also col I'ecting , data on the school program. We are trying to 
find out abOHt the instructional programs iDffered -to these students, and the 
characteristics ofT:he school personnel involved in rendering thesfe educational 
services--the teachers, principals, etc. 

Since we are going to follqw each student we must identify hini or her year 
after year without ever knowing the student' s 'name. After all, ther'e is a 
privacy law-the Buckley Amendment— whi ch prevents us from using the student's 
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name/^ All we know is the student's number. And that , means that the records 
■^and the- match between the name and the number hrf^e, to stay at the school. 
Conversely the school will n^er h^aive access to' the individual data provided 
by the teachers or the principal 6r others because we retain that data.- In 
order to follow t(ie st-udents individually, we!re.mak>ng up packets for each one 
'so'that^the teacher can f^Tl out the test and instructional information about 
every ^student. Everything is contained together in that one packet. At SDC 
we haVe to mail thesg at t4ie right times to- meet the schooTs schedule, which > 
may begin- anywhere from as early as August 13 to as late as September 20, 
These dre '^ome of thq things we have xlona during oim initial planning year. 

The second year, the year we are rn right now, is. our first operational year. ^ 
beginning this fall, in the third week of the ^hool year, we wiTl administer 
the student test materials. They will b^ shipped back to Santa Monica where .we 
will score the materials very rapidly in our own scoring facilities, and we 
win return to^ each school the achievement test results for each student so 
that tha schools can use them for their own counseling or evaluation purposes. 
During the school year we will be collecting' information about the teacher, t)ie 
principal, and the program. In the spring we will re-administer the materials,^ 
and hopefully, start analyzing our data to see 'what kind of academic rope we ^ 
, have associated with each of tHe different programs and instructional feature 
characteristics. ^ 

Itfe^'will do the same things in the §^cond.and the thi rd operational year, and if 
. Congress and God^'are willing, in the fi'fth and sixth operational year--all the 
time analyzing our d^ of course. In t^he seventh year we will- come Op wi^th 
the answers to ouMmitial policy*questions. We will try to answer these 
questions as we go along within the overall design of the longi tudinal^study. 

.The most .important .parts of this study depend upon determining a truly national 
representative sample. How do you get that?. We sentjout a questionnaire to a 
sub-sample ^f 5,054 principals of those 64,000' schools which met our definition 
of the^kind of schools we^wanteil. We asked 13 ^different questions concerning 
the size of the school, the number of teachers tJie poverty level in that 
school, the ethnic tompdsition in the school, ttie kinds pf compensatory pro- 
grams in the school.. }le wanted that information so. we 'could check against our 
idse sample of 54,000, but also so we could check* ochr longitudinal sample of • • 
^ 251, schools against the 'bigger sample of 5,^000. Over '^5 percent of the ques- 
tionnaires have been returned by now^ ' \ - 

Now the question is, how do you select' the schools from amon'*^ this large ^ 
sample? Well, we took^hese three qriteria— geography , LEA size, and poverty * 
.indtfXr-and Ri^de.up a talle with cells. ' We wanted schpol from' each of the fen 
O.E. regions. We had three sizes^of LEA*s— small, medium, ^nd large, , bel ieving 

vChat this distr^&uti on" would -be sure to s^'cure us a fair percentage of rural 
schools, medium s)Shapls, urban schools, and some large citj^ sdhools,. Jo estab- 

. lish the poverty index, we over-collected on. hlgjyDoverty level lEA*s, and ther> 
selected from some-m6diiHn-poverty levels; then.'^om the reSt of ' the LEA*s we 
"Set up 90 ceTls, and took all the schools in. the Country and put-then] in those 
90 cells,' and then drew^randomly from the sctiools that fell into* thosfe cells. 
That ought to have given us 27Q 'schools, but you wilV notice we r^e^tlly had 251 
because some qf 6ur cells turm^d out te be iero. For example, in Region Xi'II ^ 

if here is no large LEA that has a' high level of poverty: In some^plares therj 
lare no rural schpals with hi^ levels of poverty.* ]im characteristics of 'the 
schools were fedjnto the computer which gelieraited^the numbers. The schools 



selected-have been generally very cooperative. Of the 251 schools th'at we 
^identified, only l^declined to participate, and of course we just took the 
next pool on the cell. ' ; 

. Now, what about instruments? On the achievement tests we are giving a part of 
the reading section and a part of the math seQtion Df the' Comprehensive Test 
for Basic Skills. For our purposes, that was the best instrument of all the 
instruments published. Actually three or four of.them were almost the same, 
but we could get from the CTBS'.test publishers information on the ethnic 
composition of the norming groups, and wfe could do a deep biasing study on the 
individual items for that test better than we could for so?ie of the others. 

As you know, t-here has been much criticism ef achievement tests. To measure 
achievement, other than by the formal achi(^vement test, we had a panel advise 
us on functional literacy tests appropriate for second, third^'and fourth * 
graders. It turns out that there are no such tests; so we have devised a ' 
functional literacy test which the students, at least-, think is great. It has 
good psychometric characteristics. Unfortunately it correlates too highly with 
the achievement test. After all, if you. are going to be measuring reading as 
the kids actually use it out in their everyday play activities, or their work 
activities, or arithmetic, it is going to be generally the same sort of arith- 
metic and reading that they have in the achievement test. But it is within the 
'context of the playground, of the soda parlor, of the k>tchen--how you make 
Bisquick pancakes and thing? like that. 

We also have a survey of attitudes toward math, of attitudes" toward reading, of 
attitudes toward the school and toward the self as a student. It is-partially 
a standard survey published by'a psychological corporation and ■ partial ly an 
. instrument which we developed ourselves. Those- are the instruments we are 

Cng to be giving to the students, and we think they are pretty good instru-- 
ts. -The pretest showed that they worked >vell. - . • 

T*e next one, for student participation and adfivities, is the weakest part of 
our study. That particular instrument is supposed to figure out for each 
student how much reading and what kind of reading instruction or math« instruc- 
tion that student. was exposed \o throughout the year. Four times throughout 
the year we are asking the teacher to fill out this information for each 
student. . In a typical week, how ma n^ hours of reading, how many hours of math , 
instruction do the students have and 'how was it given? iWas it the teacher 
dealing with thff whole class, was it a teacher ai.de working with, that p^rticu- 
^Icfr student, was it pull-out instruction, .was it a high intensity language lab-- 
what was it? Hopefully, we will find out student by student what kind of 
In^struction in math and in reading they did get. In the pretest -that instru- 
ment worked very well in som^ of our schools, but in some of the schools it was 
not a satisfactory instrument because it seems th^t in some of the schools in 
the country, the teachers in the elementary grades have great diffioiilty read- 
ing. We've revised the instrument extensively, and we're trying it out again. 
lAs' we' get the results, we may devise another instrument becfiuse that is a very 
important part of the study. 

•We' re.asleing the teacher to fill out something about the student's background, 
the nature of^ the home they come from, whether they went to- a different school 
before they entered this school, sex, etc. We are trying to find out from each 
student what they did last summer, We want .to knojy if they went to summer 
school.* Did they go "^to a camfll^ did they just stay at \\^ss^ did they Wofk'in' > 
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the field, what did trhey do during the summer? Ihen weVe completing a ooippen- 
satory education log for each student. Are they supported by Title I or some 
other title or State education funds *or special d4^^^nct funds? 

We're also collecting information about the /flat u re of the school" stafK the 
size of the school, the location of the school, the amount of funds going to 
the school, things of that nature. The principal questionnaire has demographic, 
information on the principal, but more importantly the principal's attitude^ 
towardjiscipl ine, toward relations with his teachers, toward freedom to be 
flexible in their teaching, etc. For the teacher, in general, we are as.king 
the same questions:, demographic information, educational phi lo^ophy, proce- 
dures--what kinds of activities are preferred, how math and-reading are handled.. 

For program characteristics we are "tryi'n^to identify what each of the compensa- 
tory programs strive for, what they consist of.' We want to know if they have 
any oth.er means of assessing what actually transpired during the period of 
comper\satory instruction? Is there any defined objective?' ^ , 

When we started our study, we wanted to be sure that we had in our sample some 
schools that offered what were thought to tie the best compensatory .programs in 
the country. 'We sought a contrast. We asked State departments of education, . 
the Office of Education, and various consultants to nominate schools that they 
thought had exceptionally good programs. From- the 25,0 schools nominated, a 
panel we selected weeded tbat down to about 80 schools which We then visited to 
try 'to verify what-^nd of program they had. We now have 45 schools that 
appear to have different,' successful compensatory programs. These will be' 
treated^ust as any other part of the sample from the point of view of data 
collection /but they were not randomly selectedi--they were purposely selected. 
Some^of thenr are nat Title I schools. Some of them are in very affluent school 
districts. This s^ection confuses pepple who think ours is a Jitle I study. 
Well, yes, it is; but it also has a lot of other, little handles on 'it. So, now 
we have in our sample 326 schools- and'aboUf 120^000 students. 

'Another part of the*study is cost-effectiveness. We have sufc-contracted wjth 
RNC to do a cost effectiveness study on the compensatory programs. We will be 
collecting at tHe classroom' level information about the resources that are used 
in that classroom and Vhat the student, gets from our participaJiionHaw. We are 
going to try to relate the amount of resources used for instruction to the , 
progress made by the students. ' , 

i 

We have alsp sub-contracted with^ DECIMA, .a survey research organization in Santa 
Ana, to seek an answer to that question which Congress asked— what is the rela- 
tionship, between education disadvantagement and economic disadvantagement? We I 
want to. know what the correlation between these two is and whether to advise a 7 
change 'from economic^to educational disadvantagement in the^'allocation of funds. 
What influence will that have on the distribution of funds to the States? Tof a 
politician, this is a very irpportant question. DEGIMA will be going to 15,000 • 
households of the students involved in the study, the homes of - approjcimateT^/O^ 
students from each scheoK^ trying to find out three, different kinds of infopnatio 
What is the income and econbmic level in that home? What is the ;ntelleptual cli 
mate of the home? What is the parents* attitude taii/ard school? / 

As you can see this is a big and complex study, snd because such survcySvaie — .^^^^^ 
often critici^zed for getting s.uperf i-cial data, we will conduct a verif ieat.ion 
study by .using semi-structured techniques for *two Weeks at eacV of these ^schools. 
Hopefully, we will then be able to^de^cribe those programs and communicate them , 
ta the education commuflity. ' ^ • ^ ' 



ERJC 



^ ' "NEWS FROM THE POTOMAC".. 

. ' Harry Phillips, Legislative Liaison 

, U.. S. Office of Education ' " * . 

The educational ^peasures undergoing del iberatiort during this session ofWngress 
. deal, primarily with higher education and vocational education. All appropria- 
tions for FY 77 are under discussion right now. The likelihood of veto is 
^"^^ edul:^tionaI. section alone of the HEW appropriation is 
about three-quarters^of a billion dollars above the Presidential request. 
Generally Congress wishes 'to dispense with the measures now befbre it in oVder ' 
to take a thorough look at; future alternative roles for the feder^al government. 
In my judgment the next Congress will be seriously deliberating upon a new .role 
.ip education. Changes witl be made^n the prionty of support for education, 
and Congress will consider, maj^ alternatives to the current method of allocat- 
ng federal resources. The, categorical grants-in-aid approach has burgeoned 
intq more than the educational, community can cope with. Consequently, there 
will be some fundamental shifts in the national policy on educatiock 

We Aill h^ve many problems to work. out in relation to the fiscTT year begin- 
. mng date shift from July 1 to October 1. All of the dates for the submission , 
^ of. program reports and fiscal reports :have now been changed. In some instances 
btate educational agencies may need to keep dual sets of fiscal accounts and 
make dual program accounting arrangements until the transition smooths out a 
year from now. The change becomes critical in a few instances in which report 
dates alter ingredients of fortujlas. In the vocational "educational legislation, 
in the Title I legislation, and Indirectly in the Emergency School Aid Act- 
legislation, annual per pupil expenditures of the State and Jocal resources ' 
become a factor in interpreting how much money flows out to local school systems 
or to the State. If you. change a fiscal year from July 1-June 30 to October 1- 
September 30, you have some basic terminology .to redefine or some' basic account- 

• ing adjustments to make in terms of a record- keeping system that generates 
annual per pupil ex-pen ditu res. We hope that this data can continue to be 
collected and maintained on. Jyly l-June 30 -basis rather than changing the 
statistics to conform. to the new fiscal year.- Under the" FY shift, recording of 
program data may. become more complex, but retaining the ol,d fiscal year collec- 
tion period IS preferabl.e to changing the , definition of basic data items. 

Other Items of current interejt arfi^the educational authorizing bills thaf are 
now before the Congress-. In vocaUonal legislation there are considerable- 

, variations between the House bill and the-Senate bill. Both bil Is enhance the 
tuture of vocational education, but the way the two Houses envision' that future 
differs -considerably. The greatest difference is in the way vocational educa- 

, ,tion Should be governed at .the State level. The House projects a continuation 
of the current- governance pattern. Jn .'some States -the decision-making body is 
the State Board of Education, and in some States, it is the State Board for 
Vocational Education. But, the, House has enlarged the role of that State board 
by asking that its membership be broadened and that it contact a viider-ranqinq 

• group of people for consultation. «« y y • 

The Senate version would upset the existing State, arrangement considerably by 
introducing a new planning commission for vocatio.nal eduction— 4 coimission 
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that would approve the way a State- decides to allocate and utilize its vocation- 
al education resources that are Federally appropriated. The conpissi-on would 
not be the State Advisory Council on vocational education either; it wouftfi be 'an 
entirely new State Planning Gommission. TKis substantial change is highly con- 
troversial. How' it will be^resolved is unpredictable at the moment; however, 
most of the discussants I hear oppose the Senate version in favor of the House 
version. . 

The House version of the vocational education bill contains considerable consol- 
idation; the Senate version is a continuation p.f the existing vocational legis- 
lation.. The Senate bill also cbntains an -extensi-on of the higher education 
authorities currently in operation, but the House bill does not contain that. 
Some of these items are a bit'controversial . In addition, the President has 
conceived an anti-busing piece of legislation that may be introduced and be • 
superimposed on the Senate versiop as it -comes to the floor. With the hurr/ed^x^: 
schedule Congress is now -in during an election year, plus the fact that the 
SenS^e is caught up right at the moment on an extensive tax reforpi bill, we may 
get very fast decisfon-making and compromising on IfnisT legislation. 

The next few months are going to Jje very busy for this Congress.! The educa- 
tional decisions they make will be vitally important. There are\a few pieces 
of fringe-area "legislation that State educational agency personnel- should be 
alert to because there may be a few benefits for education. One is a bill that 
the Senate passed over the President's veto. This bill has to do with the . 
public works jobs. The potential -educational benefits are that the bill will 
allow the refurbishing, the r^uilding, or the revainping of public institutions 
that were, in the first place, built in part «Nth federal resources;, and ttiat 
provision applies to an array ()f educational buildings. 

The CETA legislation--the Comprehensive Employment Training Act--is up for re- 
authorization, and will be enacted. The new CETA bill may have expanded author- 
ity in it. Another bill, the general revenue sharing bill, will be renewed 
sometime between now and December 31. " The former version of that bill prohib- 
ited using more than ha-lf of the funding that goes to local levels for purposes 
of education. That prohibition will probably be removed from the new bill. 
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Several years-ago in the State of Nevada the Sovernor coimissioned "a studV on 
educational accountabvlity. People in communities all ^iround the State had an 
opportunity to present th^ir views on education—their likes and dislikes, 
aspirations and. hopes, etc.. The ^inal report consolidated the attitudes of 
teachers, administrators, parents, , interested citizens, and studerfts". The 
legislature then took tiimt report and asked the .Department of Education to wor1< 
with the school districts in implementing some of the recormiendatibns and to 
r.epo»t back to the legislature for the succeeding two years the progress that 
had been made in the State.- ' ' - 

Given that di^ective, we developed a comprehensive planning system model which 
we call a discrepancy-based model. The model' tiskes into consideration three 
different elements in the needs assessment: student ^eeds; "student performance, 
program or process variables in education; and societal expectations in educa- 
tion. First we establish process object>^es or program objectives, student 
performance objectives, and educational goals as perceived or translated by the 
society. Then we attempt to- measure thf present* status against those state- 
ments of expectation. By comparing the difference between expectation and 
present status, we identify educational rteeds. 

The complete model* is conceptualized as a wheel because pTanning isn't sequeV 
tial or linear; it seems to.be circular witb all the parts ijiterrelated. Evall 
uation IS' related to every segment of that wheel. In going through the planning 
process, one is continuously involved in feedback, adjustment, artd evaluation. 
The' model is based upon what we call process objective's and performance objec- 
tives. The performance objectives indicate changes of behavior on the part of 
learners; and teachers, or administrators can be learners as well as students. ' 

The strategy that .we used to incorporate this comprehensive planning system 
into an th^ federal programs administered by the department was to ask dis- 
tricts to complete a needs assessment based upon the model, and to follow the 
steps^ and the comprehensive planning process as. they developed a proposal and 
reported against it. We developed application and reporting forms that were 
consistent with th'e model. If they were interested in the planning model, we 
a&J<ed them to allow us to field test the model in one, or two, or three of the 
schdtajj^nd to give us one full day with their teachers to help get the proj- 
ect started. .We informed each district superintendent' of the ground rules. We 
emphasized that nothing was being forced on thVn, if they didn't w^nt to do it, 
they didn't-haVe t6, and no one would feel bad ihout it. We asked them to talk 
to their building principals, and if the^ were interested in having their staff 
participate in the plan, fine; but if they were not, we would go right on down 
the road. 

.We did ask the district' to make a commitment, ff they wanted to participa.te, 
they had to free their administrative staff and their teachers for' a full day . 
so that we could m?et with them during school time tP describe the process -to 
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* them. Although we would provide-the technical assistance to them throughout 
^the field. test of the planning system, the district could back out at any time 




E\ teacher^ got together and established Rersformance objectives for youngsters 
gave them four different sets of materials to look at that were really pro- 
im structures or hierarchies of .skills and concepts which had been develops 
•by otberi; but we didn't pressure them to adopt any particul^. system. We let 
them identify the hierarchies of skills ahd concepts Ihey'felY were best repre- 
sentative of their instructional program.. After a month or tw? we would return 
to discuss and critique >^hat tliey had put together, working toward the develop- 
ment of an instructional manageitient system in reading and arithmetic that reall}^- 
. became for them a new pupil-monitoring sj^stem and a new pupil-ipting system. 

' When we first met With the teachers they were confused and upset about having 
to become involved in the project, but when they started talking to one another 
and found out that some of them really did not know how tof teach reading, that * 
a they did not know the discy»ete skills and concepts that some of their colleagues 
pursued in the teaching of reading; then they began to get interested in the 
entire process. We encouraged discussions among teachers at the sa%e grade 
level and at different grade levels. We asked them to tut across grades. We 
tried to practice what we were preaching.. When we met with the teachers,, we 
would %Bt specific objectives to be met by the next time* we returned. 

'* " 

.As we tried to get telichers to look at skills and concepts and how youngsters 
demonstrate these skills and concepts, one interesting outcome was that teachers 
began to get a better appreciation of evaluation, i^ot only did thfey wanrt(to 
know whether Jo+inny or Mary had mastered a si<ill,.but also whether or not Vheir 
.colleagues were doing a good job in measuring those same skills* Now they ape 
getting interested in putting together their own homemade CRT. * 

•The instruments are not completely sophisticated, of course, but the teachers 
.are able to talk about what they want the ypungster to do. They are able to 
think about what kind of setting they can 'create to let the, youngster demon- 
strate that 'skill. And, we a|e bringing in consultants from larger school 
districts or from private corporations to expose the teachers \to more sophisti- 
cated s^trategies and procedures that they can 'adopt as part pf \their instruc- 
tional management^system. Presently we arie not moving as fast they would 
like us to. ^ ' . ^ , • 

The Department of Education is conducting statewide evaluations through con- . 
tracts -that we award to outside agencies to perform different tasks for us. 
Last year-, we awarded a needs assessment contract' for a right-to-^read project 
to three outside agencies. We were able to involve three agencies because thg . 
model for needs assessment is> clear enough so that we can cormiunicate among 
contracting agencies and .don't have to depend upon one agency to complete the 
entire needs assessment. We asked^the contractors to measure student perfor- 
mance in reading at the third, fifth, seventh j^welfth grade levels and .at 
young adult levels in the State of Nevada to attempt to define for us functional 
literacy and give us some standard that we m*ight "B? able to use in discussing 
s functional literacy with our constituents in the State. Th^n we wanted to look ^ 
at process variables and requested one, contracting firm to visit the exemplary , 
reading programs in the-^ State of Nevacla and identify for us ijie various instruc- 
tional components of those exemplary reading programs. Finally we triecf to find 
out how^well prepared our teache^s are--wliat kind of training and experience do 
the teachers who work in these exemplary reading programs have. We have* tried 
to evaluate both process and student performance this past y^ar through a. . 
discrepancy-based model , which has proved effective.' 
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ASSESSMENT MODEL FOR 
VOCATIONAL EDUCATION IN NEVADA 

^ , John Madson \ 

Iit'Nevada, we, have a Statewide planning^ system ^or setting up vocational educa- 
tion 'programs. The system incorporates five maior categories: needs and oppor- 
.tunity information, planning, student services, instruction, and educational 
accountability'. Withfn each one of these. categories are 12 functions. Every 
application or every local plan--whether the district is starting a new program 
or is renewing on an annual basis an existing program--mus< address each function. 

In order to provide the best vocational education program, planners must first 
determine what the community needs ^re. Without a needs assessment as a founda- • 
tion, the planning cjin be haphazard.. In addition, providing the community an' 
opportunity to participate in an assessment brings about positive community 
reaction and support.' .The needs assessment acts as. a barometer. If students 
are not getting jobs because they are poorly prepared or because they are trained 
for jobs which are not available, then the whole purpose of vocational education 
is questioned. 

•To avoid the man^pitfalls, in planning and admtnist,erjhg various programs and 
to k;eep abreast of community needs, I believe that the assessment- model which 
we put together for use by the local education agencies provides the kind 9f 
information needed for planning and implementing the various program^. A gdbd 
needs assessment model shpuld- define the systematic process which describes what 
is and what ought to te- The process entails populatioru«nalysis,'job market 
analysis, job performance analysis, and identiffcattoti*of resources available. 
From an assessment of these areas, the priorities, of the:program are <tevel oped 
by each agency. 

Because teachers must implement programs, their experience, ability, interest, 
and needs must also be taken into consideration. In Nevada, once a year teachers 
are asked to complete a questionnaire which gives input into the program planning. 
This Information also provides a barometer for -Ihservice training. 

An assessment of the general population requires information about' how many peiple 
live in the area to be served, what Is the age brea^cdown, what is the female- 
male breaWown, what percentage represents minorities, disadvantaged, handicapped, 
what are the housing characteristics, what is the level. of education, what 
percentage of -the population is bilingual, etc. Much of this Information is 
available ^1n various census studies, but the idea is to summarize the information 
and make it readily available. ^ , . 

Because the j^i^ market is constantly changirtg, the .local administrator has to 
keep his finger on the pulse of the employment situation; and this can best be 
accomplished through, a job market analysis. Specifically, the job market • 
analysis ,is designed to apiswer the question, "How many and what kinds of jobs 
should students be training for?" In spitp of the relative mobility of people, 
a great many students stay in their local community, and work and live there; 
S^he job market analysis of the district is extremely important. 
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» Theft we consider the area .^whether ijt 1s- rural or metropolitan. The type of 
area 'is esptecially significant in planning community college^r continuing 
programs. We also take a look at the State and national employment needs. ' , 
State ^ind national priority items, such as solar energy and air purification, 
are important factors in the planning of vocational education programs. Then 
'We consider tlie job performance analysis. 'The'job performafice analysis Is 
probably the most important element in the entire planning model. If the 
stiidents are not trained in the proper competencies, they will be unable to 

, get the jobs 1^r which they are being trained. The job analysis addresses these 
questions: Are the tasks required by the employers identifiable components in 
the vocational education program? Have existing students or' ex-students obtained 
the sequence, of skills necessary to enter the job, and to what degree have they 
achieved these skills? Is there an appropriate committee of teachers, adminis- 
trators,- and advisory members--ci tizens and students--fo review and' update those 
skills needed *in the occupation? Is this procedure carried out on a routine, 
regular basis? ' ' 

' ' ' ♦ 

A job performance analysis a-lso covers existing programs to see if they are i 
• mating the needs of .people wit*hin the district. By reviewing annually the 
national definition of the particular t)ccupation or job, and collecting data 
from employers regarding the required competencies to per/orm the job, the 
analyst can find out which competencies should be incorporated into the program. 

Nevada is moving rapidly toward competency-based education, particularly in 
^ vocational education': We will no longer graduate students or give them passing 
. t grades if they are not competent in the basic skills of the occupation for which 

they are being trained. We have' a form" to obtain information concerning com- 
, petenci_es from various employers.. We interview target groups of students'to 

see that the skills they have learned compare with the'skills that the employer 

requires,.' Finally, we do a follow-up study, tracing the' student after he 

graduates. 

The final, major category in the model is resource analysis. In this part we 
Identify the physical resources available. We also look at the federal. State, 
and iQsal finances that are available, and we make an effort to look- at the ^ 
human resources available in conjnunities— volunteer teachers. and aides and 
speakers. We encourage interest through the use of local lay people on advisory 
councils. We also consider the availability of the professional staff. •• 

This, very briefly, constitutes the needs 'assessment model -that we provide for 
all the local school districts in Nevada. When they start a new program, they 
go into much more depth than-when they continue a progr-am; but each year, the 
administrators, the local vocational director, and the principal of the school 

I » the^yocational education program and assess what the program is accom- 

•» pushing and-what it should be accomplishing. 
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STATEWIDE EVALUATrOi! OF INSTRUCTIONAL PROGRAMS IN NEW MEXICO . ' ' 

by 

Alan Morgan 

The New Mexico Statewide evaluations consist of. several measurements including 
a State-required standardized testing program and'a New Mexico objective- 
reference test program. Five years ago we went to one-third of the school 
districts in the State of New Mexico and asked teachers, administrators, and 
parents of students to help us write some basic cogniti ve-skijl 'objectives. 
• From these we derived a set of objectives that appeared to be the most fre- 
quently chosen objectives in the State. The end product of thiat survey was a 
basic Cognitive-skill Objective Bank, which was, divided into the subject areas 
of mathematics, science, soc<i^il studies and communication skills. ♦ 

Then we returned to the school districts in the State and held cbjective-rating 
workshops. In a forced concensus process, the participants were asked to make 
a distinction between those skill objectives which were of most concern and 
those which were of least concern. After, the s^ool district had chosen a set 
of, instructional objectives, it became' the responsibility of the State educa- 
tion agency to develop tests for. those various schools, based upon the objec- 
tives they had selected, in order to allow a rural school district to choose 
different kinds of -instructional objectives from those of a large urban school 
district. 

Over the last several years we* have been looking, at'what kinds of objectives 
districts have chosen, and we have^ compiled ^nd published these in a-document 
entitled Trends and Priority Objectives as' Selected by New Mexico School 
Districts . Interestingly enough, although we have found some variation in the 
kinds of objectives that are being selected, generally there is some uniformity 
in the*concerns that people have- all acres? the State of New Mexico. 

In addition, we analyzed objective selections in terml of role types, for 
example, did teachers choose different kinds af objectives than did** students, 
or did students choose different kinds of objectives than did administrators, 
etc. We found no statistically'significant difference in the kinds of objec- 
tives that were being selected by each of thoserole types. It would appear 
that there is a great deal of commonality in the kinds of concern people hpve 
throughout the New Mexico school districts. 

5lhene)^r a district chose objectives, the State Department of Education chose a 
'te?t: to measure those objectives and reported that information back to the- 
sehool. - The purpose o^ doing so was to help reasslire those who believed tests 
developed in New York or California could in no way measure the kinds of prog- 
ress or skills ^ew Mexico children seem to have or do not have. Wlrat we were ^ 
doing is sampling more iteirts in a certain domain— the division of two of three- 
digit numbers— in the objective-based^ test than. We qould do in a standardized 
test; and furthermore the standardized tests normally' don't cover the same 
areas, or at least* the sam^ precise areas, as we could cover in the obj^ptive 
based test. Ypu won't f ind too^'trwcti about tTie various-cultures in New Mexico 
in the CTBS, nor will you find out too much about career education. The CTBS 
is not sensitive to local change. Thg whole process of developing a test, 
field testing with a subset of ttte population^ revising that instrument, 
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printing, packaging, and distributing is incredibly difficult and complex. ^ If 
we could have done it^any other way* we would not have gotten into the test 
construction business because it is not somiethijig that amateurs do* very wel?. 
We've termed these objecti ve-^ased tests rather than criterion-reference tests 
for the simple reasorl that we did not require a student to meet an objective by 
completi-ng X percentage of items correctly. 

We also have dealt wilih regression analysis for the last four years, and we 
report back to the scjiools howi they dixl as conjpared to some sort of a range of 
expectancy -as computed by the State department^ of education. We use three 
factors to compute that expectancy as a pre-primary irput factor to project some 
sort of a range of performance of each school in- the State. >le use previous 
test performance, medfan family income, and student mobility. If the school 
consistently scored below that expectancy, then the State department pf educa- 
tion would become more" involved with that school. So we are using t(ie system 
not only as a way to identify trendsr arid priority objectives but aVso as a way 
. to deploy staff of the department of education. 
«^ » 

There are other facets of the Statewide evaluation program, including the 
reporting. of college entrance test scores. Although these are not part of the 
State- required testing program, we try to provide some information about why the 
scores change as they do. . • 

When we first started the standardized testing program, teachers often didn't 
get a chance to see the teacher's manuals that came with the tests--or if they 
did, the manuals were handed out at the time the test was, given and t^ken up 
" , immediately afterward. So we developed a New Mexico teacher's manual for the 
New Mexico classroom teacher./ The manual deals with questions like, "Why are 
, we doing this? What do* we get back? 'How can we use this information?" Any 
teacher who gives a Statewide test gets the teacher's manual to keep. 

Teachers are probably not too excited about having to give a test from which 
they do not get back much information directly at the classroom levtel. Now 
that the evaluation program is becoming more a responsibility of the local 
school district, we assume that there can be .a mucfi* better appl.ication for the 
classroom teacher artd maybe for the individual child than in the past.' 

Five years ago, educators in New Mexico at almost any leveU-classroom teachers, 
/ teachers' aides, administrators, superintendents of various school districts- 
were not really thinking very "objectively." • Most of our program planning was 
^ somewhat primitive in terms of the Min.imug Standard for Public Schools vn the 
State of New Mexico , a State board document with which all schools have to 
comply. The Kinimum Standards essentially t>oil 'down to three stateme^nts: the 
schools are required to assess the n^eds of their students-, to plan ajDrogram 
to meet those needs, and to evaluate that p^rogram. Minimum standards imply 
some regulation of -schools and districts to insure "quali/ty" education. What 
is quality education?^ What are tfiose basic skills that everyHcid should have? 
We' re* groping more with that problem than we have in the past. With the -move- 
ment of the objective-based testing program to the local level, we are nov*^, 
considering a minimal-skills assessment program^ although the State Board is 
undecided on the question bf minimal skills or minimal competency testing. 

In New Mexico fifty or more percent of the students are children who have an 
*^ ethnic background different fr^m the majority white community. In testing 

programs there are no special exceptions for the bilingual chiild, exclusive of - 
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tjhose that are set out in the regulations; that is, .if a student cannot deal 
with the' English language, if that's not^ftis dominant language, 'i/ he doesn't 
understand the testing situation that he or she has bee^i placed fn, then the 
. student is not required to complete the test. General guidelines indicate th^t 
students who are in the mainstream programs 'are to take the tests at the grAde 
level as they are required. If the student cannot^eal with the testing situa- 
tion' because of a linquistic variation, then we do not require the student to 
participate. ' Usually that doesn.*!.;^ happen very often. . . 

Unfortunately, even though we haye had biHngual programs in New Mexico for a 
considerable >ength of time, we still do not have an abumlance of non-English 
materials for a number of reasons.' There are .va*:iations of dialects across the 
Stale, and some of the Indjan- languages are not in^Vritten form. Nine percent 
of the population 1s Indian, which constitutes a fairly numerous group of 
students. We have in recent years conducted only an informal evaluati®r\ of 
bilingual education programs in the State. ' 

• • * * . * 

The needs assessment process that we went through for the entire educational 
System began at t+ie State level and moved to the district level. When we 
'started developing objectives and asking districts to make a decision about 
which objectives were more: important, we were getting into the assessment . 
process. We've come to understand in the last couple of years that the process 
needs to start at the grass roots level instead. 

Because the new minimum standards, passed in. July of this past year, require 
schools to assess student needs at the school level, we put together a process 
guide for thenr use. They could use other kinds of models for needs assessment, 
but we simply wdjnted to provide some examples of some of the kinds of. instru- 
ments that might be helpful. In essence this document offers a way a school 
district can identify student needs, and that' s" real ly our basic philosophy. 
From that determination will evolve other kinds of needs--staff needs, program 
needs, facil ity needs, perhaps even policy needs. 

Two years ago a State mandate required that-by 1977-78 every school district in 
the State would offer a State-funded early childhood education program. '^It was 
not a change of the compulsary school attendance law; we didn't sey every kid 
.of a certain age had to be in the program. We simply said that every school, 
hfid to offer a program different from kindergarten. Our intent was not to have 
d mashed-dowh first grade. A rather -rigid, classical instruction design woiffd 
not be appropriate for the five-year-old child. We wanted to emphasize social 
and emotional growth- more than intellectual or cognitive growth. We have als9 
prepared for schools a document dealing with the evaluati'on of early childhood 
education programs. 
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EVALUATION EFFORTS IN THE STATE OF TEXAS- 
CAREER EDUCATION, DEVELOPING READING AND MATHEMATICS 
OBJECTIVES, EVALUATING COMPENSATORY EDUCATION 

by 

Dr. 'Waller Howard 
Carl Defibaugh 



Editor's Note. : Dr. ffmJard discussed briefly Texas involvement in the develop- 
mnt of Essential Reading and Mathematics Objectives and the Texas effort in the I 
assessment of career education. Both of these efforts hav^e 'produced publications . ' 
Essential Heading and* Mathematics Objectives is currently available from the 
^Tes^sT^cation Agency^ and the project report^ Career Education Measurement 
Series (CEMS)^ will be published shortly. 

Mr. Defibaugh discussed the Annual Report of Programs for the Disadvantaged in 
Texas (Title I Regular^ Migrant^ and State^ Compensatory Education). He also 
reviewed briefly the ^following evaluation materials prepared by various school 
districts and the Texas Education Agency: ^ 

From ^Austiri' Independent School District: - ^ 



1974-1975 FinaT^ Report : Individually Guided -gducatioo (IGE) Program 
itTT- TT Tinal Evailuation Report , ESEA Tftle I "PFoject 
T97?'-T5'75 Final Report: Cook El ementary School 



V 



yOT-75 Final Report : " ESAjT l Pilot Pfoject Assist 
^nalTva 1 ua ti on Repor tTnE"SA?^ Bilingual/Bieultural Project , 1974- 75 
PmaT EvTroatTon Report :' ESQ" Title VII Bil inguar"Project , 1974- 75 



[nalTva luation 
inal Evaluation 
School Based ^Plahning- Manual CiPT) 

From Dallas Independent Scho^ District: 



Measurement Profiles , 1975-1966 

A Summatry of the Major Findings from " E^igrybody Knows What Profit Is^" 
TTbstracts of Research anct Evaluation Reports , 1974- 75 ^ 
A Plan for the Use of^ Ri?earch ; Evaluation , and Information Systems 
Resources , 197F« 7g" ... ^ . , 

From Fort Worth Independent School District: ^ ^ — » 

Short Term Assessment of the Middle School Plan 
InitiaTl^sessment tff ' th£ Intensified Learning Plan . 
Short-Term Desegregation Effects : TFTe Academic Achievement of Bused 
ftugents7l97l- /2 

1974- 75: The Evaluation of Six, Kindergarten Curricula ^ • ; 

Abstracls ," Tg74- 75:> Degrtgent of ResearclT and Evaluation 

A ComparisorToT ^oafiniatos'Pr jogre ss" and "Trail i tionar * tl^sses i n Three - \ 

m T Mjddle Sch^s~ ; ^ ~ , 

Final Report of the Title/ IV.. Part A: American Indian Needs Assessment 

and Referral, ^gram , 1974 - 1975 » 
V\ssessment of tffe Utilization' of the Time-Ou t Structures 
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From Houston Independent School District: 



The 80- Wnute Period ^ * 

.ESAA Basic Rjrogram : Fjna'l Re port, 1975- 76 

Contemporary Learninq~Center Final Report , -1975- 76 . 
. Community High School Final Report , 1975- 76 

An Educational Program Audgt for QIE.-MS Programs' Final "Report , My 1976 
Mdendum to Final Report : AuHvtors^ Objective RepttrtsToTI^ Mst jul y~I^ 

- » , 

From Texas Education, Agency: 

Sel-f^ E valuation Guide for Local Districts for ^oceitional Education of . 

Handicapped Students . . 

The Development of Evaluative Criteria for Training the Handicapped : 

A Summary^oT Project ^Activities ^ ^ , ^ ^ 

GuiHe to Proposal Evaluation for Vocational tducation of the Handicapped • 
Te>ta^ Education Product St"u7y : Digest of Final Report ~^ ^ 
A Model for Evaluat^ing Programs Vpcatiorlal Education for th£ Handicapped 
Evalugition of Coo^dinate'd VocaTional-Acadenfic .Education j_n Texas * 
EvaTCiation of Career Education Projects in Texas : Houstpn , HarlandaVe , 

and Fort Worth Independent School Districts 
Special Prjpject- Career EducTtion : , Texas Education Agency • , 
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/ -UTAH'S STATE ASSESSMENT PROGRAM \ . - \ 
DaVid Nelson' ^ 

•The 1975 Utah Statewide Educational Assessment was developed and refined as a 
joint effort of the Utah Sta\e Board of Education and the Institute for Behav- 
ioral, Re'search in Creativity. The study was under the general direction of 
Dr. Don K. Richards, AMnistrative Assistant and mmning. Dr. Dav«E. NeUon, 
Speei|ilist in EvaluatlH; had specific responsibility for the project within the 
State School Office. Dr. Robert L. Ellison directed the project' for the Insti- 
tute for Behavioral Research- in Creativity. . - * 

The 1975 Utah Stastfewide Educat»al Assessment was .Utah's -^first compreliensive 
educational asses'sment. This sWtewide program of educational assessment was 
condac.ted because' Utah educators are very concerned with improving educational 
practice. StSitewide assessment is- one example of the wpunitment of teachers 
,and administrators throughout the State, locaT school districts, and the Utah 
State Board of Education and its staff to provide a quality education for each 
student. within Utah' s*public school system. * ' ^ 

/The study was based ^n the goalj afJproved by the^State Course of Study Commit- 

4t'ee in the Fa'll^of 19/2 for public education in lliah.* A^tJects of ftve of these 
goals were ;measur*ed !n the 1975\Utah statewide Educatfonal Assessiient; ,lnteT- 

•lectual Maturity, EmotioQal Maturity',.Socjil Maturity, Aesthetic Maturity, and 
Productive Maturity. , The students'- tnjoj^nt of School was also assessed as Jt 
was'^'mplicit in all of the Utal) Educational Goals. Besides measuring student 
outcome in thes6 arfeas, the .study meafsured many factors whi-ch might Influence 
student achievement, 'including demographic characteristics, such as socio- 
economic back^und, ethnic group. membership, and ^sex; a^^will a^ schoo^l clas-. 
sifications, sUcJa 4s school size and Ideation. Another extremely rmportant * 

.part of the studj was »tbe, measurement of -key aspects of the eck(£ational process, 
including measures of tec*iniques-'and ^tratiegies used by educators to help stu- 
dents learn, for exaijiple, CI afssroomT Participa1iftt5n^|§evel6pmejit of Career , ' 

*Tal,ent^, and Expectations. / \ 

* * • * ' . ' ' 

Th6 1975 St^tewide^A^sessment ef/fort involved the testing of random^'iamples of 

both elementar/ and^secofidaVj^ schools. Over* 4,000 5th grade students in 67, 
•elementary schools «nd over 3,000 11th grade students^in 26 high schools took ^ 
part in the study • • . ^ • . ^ 

• . • • 

In the Intellectual - Maturity goa'l area, .Utah students were s^uperior t-n mathe- 
%atics apd scienc6. achtev-emei^t, when compared to.a;nati(yhal norm group. The 




' In tlfb^ ^motionaj >1aturity goal -area, Utah students had generally -high Jeve?s of v 
positive ardj^igifrent on each of the scales in this ftudy. The chi-ef/area of. * 
conceVawas the lower. leVel of Academic Self-Concept reported by^th^lth tirade / 
students when compared to 5^th grade students* -^-ThiS si#ggests that S^rcific 



instruction?! strategies aimed -at enhancing levels of Ac"ademic Self-Concept' of' 
ffll students, with f)articul.ar emphasis on seconjdary s'tudenfs-, Should be'inclyd- 
ed in the curriculum. , • . • ' 

In the Social Maturity -goal area, fewer than 20 percent of the %tah students 
sampled at each grade level indicated they felt they were either; "a little weak" 
or "rather poor" on a variety of peer reiatiOas activities,, thus, the great • 
majoirUy of students ^it both grade levels i-nblcate'd their .competence in dealing 
with peer relations was at least "very good.i'- The results also indicated that 
more opportumMps fo^ Utah students to engage in leadership experience are 
.,need«dj Also needed arei more opportunities to engage "-in teamwork activities 
and other group aoti«ies. ' ^ . ' " 

In thtf Aesthetic Maturity- "Ifcl area, 11th grade students generally performed 
at hiiher levels than the 5th grade students. However, performance was not - 
• high-ht either grade levelc The State "School Office Art and Mus'ic.Special ists , 
who agisted in the construction ofHhese «c'ale§,|g|t that- ideally the cqnten^ 
mea^ur^d by thesa tesj;s Should be-mas'tered by stu^^clo^ to the terfhination 
of the students 'lajj-ementary school years'. Mych cou»f)e done to. enhange^stu- jii 
dent ^chievement'Tn both the. viisuaf arts -and music.- ^uch eVforts might consi^ 
of several different specific strategies, including empha^i^ of pre-seWice and 
in-'service training, innovative 'programs, greater use oT*cur,riculum guides',* etc. 

In. the ProdMctive Maturity goal i^a, the standardized Career Planning Knowl- ' 
edge test showed that Utab 11th grade students scored about the saiue as the ' . ' 
national -Average. Aootljegp important -measurs of stude^lt's Productive Maturity 
was a scale cal'ljsd Independent Development.^ This , scaya.measured the extent to 
which^st^dents participated in se-1 f-in'ifciated learning activities-whi* 
increased their general conipe!^nce. In, general, on. selected activities , 'utah 
.5th graders reported higher levels of independent ^1 earning llhan did llth'-^rade • 
students. , The" present* thrust in th^e Stafe. ia the area of cjr^r education. 
. seems warranted in view of thes^ results. -Certainly, Utah stulehts could- " " 

b.enefit froiii .increasing fam>liarity with the nature of the world of wor*f, "as - 
. wel.l ^s' addltjibnal emphasis on actual jt* experience. Many aspe"cts of career 
-'education migtit well be jntfegrated into^lmost every segjnejit*of the existiW 
•curri'cu'lum'. ' ' •. ., A , ■ 

»F6r Enjoyn^nt of School , reasonably- positive-results were obtain^a'f both-', 
grade J^veTs,' vrfth the 11th grade students sco^'ng slightly highej^ than the * ' 
•5*h grade students^j^owevefl^, the poss'ibility existed that this result was due , 
to the. attfitiomol^tude'nts who di si ikett school. Ther^'ji gniij^ant relatiotf- 
ships of this scale to .educational process measures suggest that -broader ' 
implementation of specific 'instructional 'strategies Would 1§ad'ti> l^igher level^. 
of Enjbymerit'of Schoo].. . • . ''• ; 1 c ^ ' ' ' 

lit r * 

Th^esults .obtained from the -educationa] process measures ^intluded in thi^ . ' 
study— Individualization -of Instruction, Reinforcerpent of Sel^-Xoffcefft„ Class- 
; room Participation.-fi^yelopment of £areen Talents, and ExpectatiQns--indicate 
that such processes warrant inj;feased levels of application i^MJtah schools.' 

An ?i-mpoTtant*fimJin9 of the study, and one Which has been found national Iv^. was' 
the impact of soc'io - economi'c status on the* achievement and attitude* of Utah 
studefnts. This finding suggest^t'h&t steps need .^tc^be -taken to deal- with , the 
^^cational ^disadvantages ^h^d/^ accrue to -students'- Trom lower socio-economic 
W:kgrounds. • * ' 
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COMPREHENSIVE-SYSTEMATIC PLANNING 
» ^ -IN WYOMING 

^ by / 

Thomas 0. Kraner 



Educational evaluation in, Wyoming is directed> toward the definition of school 
goals, assessment^^t)f the school environment, assessment of the children's 

. response to the environment, feedback of information for relevant decisibn- 
' ^ .makers, 'decisions for modification of the school's environment and then a 

" recycling of the whole process, PhilosophTcally , much of Tt is ffecused at the 
.local level.^ Ve place"^a heavy, ethphasi^ on "all evaluation activities as pro- 
viding the* focal district with some kind of Information upon which they cart - 
make 'a d&cisior^, look iar a direction, or improve somethij 

The-Wyoming Comprehensive Systematic Planning Requireirent, which becomes manda- 
tory this coming September, is Simply a minimum standard under the Wyoming 
, ^ school cdde. It is not imposed by the' l-egislature, Tnb requir0ment*tates 
thdt local d*istricts musf conduct needs assessments, theKhave.to be able to 
demonstrate community uivolven^t in planning, they have to^shq^^ lhat they have 
. evolved procedures for resolv^ identified learner needs, and^ they have to 
evaluate. The requirement allows districts to pursue planning as they see, 
fit--with'State department assista^c^, employment of outside consultants,, doing 
it on their own--but to have their plan approved by the State department. 

In our heeds as^essmeHt, as it is' currently stpictured, we start with broad, 
basic community involvement, having the districts that we work with form a 
ne§ds ^ssfessi|iient commit^rfe dhd a steering committee to monitor the procedures, 
^ . We have been using as tfie^basis of the needs' asjessntent the Phi Delta Kapp'a 

- * .goal statlments for education, l?ut in addftion to these we ask for identifica- 

tion of learner needs, and let other, needs evolve froiTthose, We don't require 
'testing as an initial step in need^assessment. We encourage commi*fMt>. input 
with regard to the goals viewed as being important for edutation in theylocal 
di^strict, and then an estimation of how. the district is perceived as ctm^ently. 
meeting those goals. Wherever there is gap between high' value of goal and 
low rating of fulf i ll^ment in* the district, that becomes a need; and at that 
'potn^, the district iS asked to^dooument the need in more .detail, not only by' 
.X using information from community surveys but also by using any school data that 
*is available, Th6 State Board gives each district discretion to modify the 
; . systematic cdmprehensive .planning model. The only requirement Jmposed is that 
; Hf)lanning be systematic and comprehensive, » ' ^ ^ 

in the planning model- the f^rst step is t^he completion of a needs' assessment. 
The next^step we suggest is that the school district take the results of the 
I -weeds assessment and further defing its problem, AnaTyze the needs assessment 
, and generate some alternative solutions to meet the nfeeds identified. Devise ' 
1^;^- aft- evaluation scheme for assfessing ^the extent to which the district has met the 
.'tfJI goals they have set for themselves:,' • » • 

l^f ' ■ ^ - 

j-i We have been if^orking with the di^^tricts, offering technical assistance. Where^ ' 

- J ^a given identif jpd need requires id;ne kind of expertise we don't have, we have ^ 

''been drawing on the subject-matter consultants and the/instructional division 
for^help, but wg haven't been providing special ironey. ^ 
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Another face.t of the entire planning requirement is. tied to the accreditation 
process of loca\ school districts. We have started accreditation visits which 
not only involve looking, at those items which come'as a part of our regulaV 
accreditation ujrogram., .teacher certification, and other r^elated concerns; but. 
as a part of that overall look at the district, we have been re-examinfhg their 
progress in meeting the^planning requirement. We do get a little internal 
criticism about that .and have- some reservations about.it, but most districts 
s^eem to be cooperative. " . ' . 

I 

Another facet of what we are involved in is, of course, evaluation of federak?fy 
funded programs. I was' rather encouraged to hear our keynote speaker suM^st 
backing off from a rigid position of objectivity. Our efforts in Wyoming are 
certainly not solidly objective. We can't point to any evaluation activities 
we are involved in that center around sophisticated research designs. At the 
same time, P think we have begn rather direct^and honest in using program 
objectives developed by the local district as a basis for evaluating programs, 
and we generate our questioilrtai re and student interview guides^ ^nd staff 
interviewM'nstruments from ol^jectives for a given program and go in and collect 
data Snd cWrect our generaj izaflons tm positive-areas and areas of concerns. 
Recommendations we make Srg based on What a, given district's project said it 
was going to do in its project objectives. I /suppose, some peoole would call 
our efforts .goal-based evaluation,, and at times we question^lf maybe even this 
isn't being 4 little dishonest because frequently as we" look |t a* project we 
see side issues that don't come out of the objectives slated by the project. 
We are encouraged by the fact that With som of the programs i ke. the Title IJI 
programs, some of our recommendations have been taken. to heart an^d some effort 
has been made to steer the projec;t in a new direction". So we feel ,as if we are 
hav-^ng some effect and doing more than^just orovidvng a written report to the 
federal program director responsible for the project. Of course,#I think that 
is what irhe. whole thing is all about. , 

As a part of our evaluation activities^ w have beeft , working .wi±h -districts in 
the area of staff evaluation. We haVen/1; been writing their stfff evaluation ' 
programs for them, but we have pris^hti^ig districts, with a model which 
focuses on self-evaluation techniques and job targets'; and we have bejen working 
with them to show them t'he^advan^§i^ and disadvantages of evaluating staff ^in 
different ways and tying this to a fesponsi bil-ity fc^r not only evaluating sta,ff, 
but also building ,>n some kind of a support system to help'them improve. 

One of the big concerns expressed by districts, especially in those'first few 
orientation sessions when we start talking about the planning model and we get 
dowa to that evaluation component, is' who i*s evaluating and what? What are you 
gfiuig to do' to us, if we fail?:^.We have 'been trying' to reassure' them that we ar.e 
not passing or fail.ing them on what thftir ev^ation data shows. We .want them* 
to become if ami liar with the process and.aSe it and use the results of their 
evaluation to make- their own decisions, Jkft many of them are certain we are 
going to penalize them in some way. • ^ ' 



• ' EVASWTION OF CAREER EDUCATION IN ARIZONA 

by ' 
Richard Ruff 

• . . • ' ■ — , ' 

Let me review with you some conslcleratlorrs In developing career education c 
evaluation^tapabilities at the State department level in Arizona. We analyze 
management data,, student outcome data, and data concerning instructional strat- 
egies. We have it done by external ev^ljiators. We tr*/ to determine whether 
the ki-ds are really learning anything or not.,. We also try to establish the 
effectiveness of the different approaches that we are using in career education J> 
We have done that in a number of our* projects for a .number of years, but we do. 
not do it in every project. We#!mply can't afford it because it Is fjeirly 
^expensive. For example," in Pema County, which is our largest project funded— 
$600.000— we are putting about $35,b00;into the evaluation. 

Another type of evaluation we are doing is the Sfenior Folfow-Up Study. We use 
a different kind of form, specifically geared to try to' tap into things we are 
concerned about in career education. We glve it to all the students ra.ther 
than'just vocational education students.. We are trying to determine whether 
the students have achieved any of the outcomes 'we are after and whether the ~ 
strategies we are using are effective. \ 

Our Annual Statistical Report is a report we give to the legisl'ators . It's ^ 
simply number»data. We also give them a synopsis from the report. Our Annual 
Evaluation Report is a report that each. of our projects must submit at the end 
of 4rhe year. They have to state each goal, each objective/ each evaluation 
procedure and include their findings. In.most cases, th^t is 'evaluation they 
do as' part of their projects; it is not done by \x\ external group. 

♦ ^ * * 

We have',also developed a needs assessment systiem". It's essentially a service ' 
to the projects, which gives them an opportunity to find oyt what the career" 
education priorities are, in the community. We will help them at 'the State 
department Ui develop the foaitt, we do all the data processing, fand we give/^ 
them ba'(;k a computerized repoflii It's a s^'stem that is very flexible and j/ery 
"Mmple, so that most protfyc^s can dq .it on their own oiTce we give them sorfe 
initial instruction. Since we spespt a lot of money developing curriculum in 
car^r education » we obviously attempted to assess th? effectiveness 6f the 
curriculum. We ^id pilot tests, we did field tests. We then-went Into what we • 
called implementation testing-to find out the eff«:tiveness of those units. * 
Now, whenever teachers want to use a unit, thfe evaluation- report tells them how . 
well the kids lUed it, how well'the teachers liked it^ how well the students 
did on the perfoi^ance objectives of that unft, and how much it .cost to implement. 

We are »lso do4ng what we call "listening ^ssions." Essentially this is an 
opportunity fApeople in the State department, mainly the director and myself, 
to go out andjially talk 'to teachers Ind administrators and project staff to 
find-out what-Ts going on. . They are structured interv.iews. We ask standard 
•liuestlons afld- then just shut up and listen. This helps us in trying^ to plan 
for the subsequent years. 

One of the biggest problems I find in evaluation is that most administrators do 
not understand evaluation reports. • Part ofmy-joB^is to serve as a translator 
between the evaluator .and the .administrator. We spend'a long time working with 



our extjernal evalwtors to oiake sane that they are responsive, to the adminis'- . 
trators, responsive to the kind of tilings that the policy or governing boejrd of, 
the pypject can understand./ , . . . - 

* • ' 

In some of .oirr projects we found'put that .we weren't .hitting the high^ schooTs-^ 
nearly as much as the elementary schooU. * We were^utting a dispropoVtJfbnate 
amount of our staff tirpe into contacting people in the elementary schools for 
the simple reason that elementary schools, tn general, tend to be mbre recejytive 
to the concepts af ^areer education. Our project staff was going to those\ 
places where people loved them. 'They were welcomed there, so they went there 
more* They did what they liked to do. We needed data to give us a snapshot of 
how our staf/ was spending its, time. For example, if the project had a major 
thrust in p^arental involvement, and our staff was not-talking to anybody about 
parental involvement, theh it is v6ry unlikely that that iS going to get done. 
So we obtained a.jimple kind of management data on what our staff js doing. 

We have also obtained data on the major thrust of speaker^* visits to the • 
classroom. Did they' deal wiUi the area of career awareness? Did they deal 
with the area of self-awarene?s? Whicti of those are^s did they deal with? We 
found that in elementary schools 73 percent of the guest speakers were dealing 
with the concept of career awareness to the exclusion of other things. So, the ^ 
projects made some changes to try to healp guest speakers get involved in other 
kinds of topics. We put together, for Wample, a little broqhure on "Tips for . 
.Guest Speakers." The fallowing year, *7Ar75,'We corrected that imbalance to 
some degree. \ ' • 





/James Morter\sen, Colorado, and Helen Warner, Idaho, made presenta- 
tions on their respective States. Due to' mechanical problems with 
the tape recorder and insufficient time for duplication of additional 
speeches, we are unable to' provide their presentations. ) 
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INVOLVING ELEMENTARY AND SECONDARY 
TEACHERS IN m^ UTILIZATION 

by , 

I ' t ' 

Bin Connett' Montana 

.Lately many people are critical of policy makers and decision makers to |js£ 
jresearch data. I tend^to react negeftively to seat-of-the-pants decision-qjakingr 
birri don't think we can expect people to use data intelligently until they 
have had some experience and ^||e training in doing it, 

I would like to propose that we place more emphasis on teaching peop.le in tf\e 
^elementary and secondary schools how toHjse information. We cannot expect sohie- 
one who hasnit had any real training in tlje use of data to use data suddenly 
in 'a rational manner. I don't think that the evaluators and the people who 
produce the data can do it all themselves, and I don't'-*hink we can place alt 
the burden of changing on the ev^luator, although I do believe that part of 
our job is to try to train people in using information. People frequently 
.oversimplify issues^bec-ause th^y want simple solutions. That is what the); have 
been used to; that is whaj is easy. They need to be taught to use 'complex 
information in making rational decisions. 
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EVALUATION ACTIVITIES OF Trtf PLAijNING, RESEARCH, AND ' 
EVALUATION SECTION, OKLAHOMA DEPARTMENT OF EDUCATION 

• ' by 

Charles' Sandmann 



In Oklahoma, we have an expanded view of evaluation. We know that, technically, 
evaluation is a decision based upon a- comparison. But, we have learned that 
assumptions cannot always be jpade about the process leading to evaluation^ Our 
efforts toward evaluation have, been influenced by the impact of federal programs 
as they operate at the State level and in the public schools. 

Our activities can be grouped according to the services performed fpr the 
department and thos^ for public schools. For the department, we have produced 
the needs assessments essential for our State plan and the program reports that 
describe federal programs in our State. We also'have been instrumental in the " 
evaluation of workshops conducted by lections of our federal programs division. 
Perhaps our most important functipn, and the one which has made the greatest 
change in the use of evaluation, is staff development. Several years ago, we^ 
introduced management by objectives to the department staff using materials 
fa)m RIPP Conferences. • We participated in Evaluation I from UCLA, CSE, and 
then conducted the workshop for our staff and for LEA personnel. ' 

Accountability has been the second largest impetus to evaluation in Oklahoma. 
Four years ago our legislature passed a resolution directing the, department to 
implement and monitor an accountability program. ^Jhey may have had other 
things in mind, but we designed a three stage process^ based upon systems 
analysis. Our experience with MBO provided the basis for our planning." We 
conducted in-house workshops and then expected the total SDE staff to provide 
leadership to educators across the state. We produced materials, a slide tape 
presentation, and a video tape which was used for a simultaneous, statewide 
workshop foi^ educators. It will come as no Surprise to learn that we met with 
,some resistance to the imposing of MBO from the top down. 

Our next thrust. grew out of our experiences with accountability.' We realized 
that the use of evaluation requires a change in people. To cjuote John Goodlad, 

."School change is people change." We selected a people-change technology, 
Preparing Organization Development Specialist- (PODS), from Northwest Regional 
Lab. A cadre of leaders 'was developed, and now we are providing the service at 
the request of our staff and public school staff. It is too soon to describe 
the tffect "of OD in our state, but we hope to devise a method for evaluatlonr— 

. although its developers admit that evaluation has been elusive. ^ 

The schools in Oklahoma most receptive to both MBO and OD are the schools with 
federal programs and specifically innovative programs. We provide the data 
treatment for those who request jour services, and these reports are ^he only 
ones that could be labeled "evaluation," eyen , though they belong in a compari- - 
son phase--they comjp^e measurements inade of the project's objectives with a 
standard defined by the project. Evaluatlvie statements ar^ the prerogative of 
project personnel and the on-site evaluation team. 

Finally in our classification of evaluation services, j«e have placed the surveys 
conducted at the request of department or LEA personnel. These are usual Ij^ 
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designed to jneet a specific need or provide information about a question posed 
by the person requesting assistance. The following list of publications 
includes examples of each activity; most are- available for your use. 

.An Assessment of Educational Needs ' for Students in Oklahoma , Phase II, 1970. 

An Assessment of Educational Needs for Students in Oklahoma, Phase III, 
Vol. 1, and Vol. 2, 1973. ' ^ 

Oklahoma-^atewide Assessment . Needs Assessment, Phase IV, 1974.- 

Migrant ' tgucat4on Handbook . 1975. . ' 

Survey of Migrant. Education ir> Oklahoma , 1976.' Set of survey forms. 
^Survey of Migrant Education , Final Report, available October 1976. 
. - Anrtual Evaluation Report of Federally-Funded Educational" Programs , FY 1975. 

Evaluation of Education/Fair , 1975. 

Evaltidtion of Education/Fair , 1976.' 

Evaluation Report for Emergency' School Aid Act Workshop for Oklahoma E.S.A.A . 

Educators , September 27-28, 1975. 
Evaluation Report for Value Clarification Workshop for Educators in South- 
' eastern Oklahoma , November 8-9, 1975. • 
Evalaation Report , Value Clarification Workshops for Educators , Southwestern 

Oklahoma , April 8-9, 1l97&^ ' 

State Department of Education Goals for the Middle 1970's . 

State Department of Education Futures Study , ■1976. ^ 

Evaliiation Workshop I, Center for the Study of Evaluation, University of 

California at Los Angeies. 
Needs Assessment , 1973. 

handbook for Objective Writing and Program Development , 1974-75. 
Evalugti'pn , Book I. 

Worksheets for a Workshop Presenting Book I. 
Affective Measurement', Three Scales. 

Group Process Skills, Northwest Regional Laboratory, Eugene, Oregon. 
Arts Communication Training , A Workshoji for Training Planning Teams for Arts 
in Education, 197,6. • 
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INFORMATION UTILIZATION IN DECISION-MAKINS 



.by ' \ ' . ■ , 

Dr. Roy forbes. Director . ^ 

tiational Assessment of Educatlbnal Prograrrfs 

Education Committee of States ^ 

Many times we try to think of "utilization" strategies to make sure that we take 
the information we have generated and apply it in such a way' as to re-plan or 
re-cycle our educational programs. Whenever we start thinkiVig about inforradtfon 
. utilization, the first thing that comes to mind 1s the oroblem of communication * 
between those of us who are responsible for gene rioting information and those of 
us who are responsible for using that information in making decisions. • 

Systematic decision-making does include several steps. The first, obvtously, 
is to define the problem. Everyone claims that step is rather obvious, but, I 
know many times groups who are in the proces* of.making some type'of decision 
only talk about the problem in v^rv general terms and never get around to say- 
ing just exaaiy what the probVem.ns. 

Once the problem is defined, then you have to generate a pos'Sitle solution ok 
possible solutions to that problem. I personally .think that you need to do the 
latter— that is, come up with' a number of alternatives so that you can the"n 
mull over each one. Do something like identify financ'ial resources or personnel 
resources that you would require in order to implement one of those possible 
solutions or identify any problems you woul^ have with .ifflfilemen ting one of them-- 
trWto 1(^entify the constraints— try to identify possible ^ways of removing those 
congtfaints by working around them. Then look at the information you have at 
tha)t point,Yeviewing. the possible solutions for potential payoff— which one 

"-wdl have more bepeficial effects than the othfers. You have to continually go 
back to the previous step becausfe usually what your head is doing is synthesiz- 

-4ni^ combining those possible solutions that you weighed the whole time; and 
each time ybu do^that, you should go back and create the information needed 
for 'each new possible solution. At some point you. have to make a selection— 
the last step. 

These are, very briefly, the steps to follow .in trying to reach a decision, in 
trying to solve a problem. Each step requires certaip amounts of information 
selection. Sometimes we can sit down, when we start, and find all the informa- 
tion that we are going to need ahead of time; but in most cases we need to go 
back and work through each step, identifying additional information, which we 
cafl then use in this systematic process. 

Sometimes the best decisions you reach are those decisions you make in the • 
showijj^or driving to work, over a drink, or dancing with your wife— they just 
pop iWto the mind all of a sudden. When you stop and think about that partic- 
ular decision, oftentimes you can stick with it; I call this "model the Ah hah!' 
type and make the blest use of it I can. Now some people might say that is being 
very unsystematic, I really don't know because \ don't know what's happening in 
the brain at the time that these Ah hah'e occur. \Maybe a thousand years from 
now we will know how to analyze thinking better than we do today, and someone 
will be able to determine a more systemati;: process for that, . 



One thing you can do which ties In with utilization of decision making— when- 
ever you have one of those Ah hah's, sit down and try to reconstruct what 
information you. believe you have used that led you to that decision.. I think 
that's the clue. Many times we feel that we haven't used 'information in a 
very systematic way in reaching a decision that way. Sit down and try to 
figure out all of the processes that you could have brought into flay to 
reach that decision, and yog will start identifying information that yUinmay 
have subconsciously used. Then you can identify those pietes of information 
and, consciously us€ them. 

I think we need to develop a respect for information which -appears in a dif- 
ferent way from the systematic way. If you do not go back and look at the 
information that you used, maybe subconsciously, or look at the information 
based on experience, there is a danger that you will think you really don't 
need other information, and it creates an attitude about the use of informa- 
tion that I think can be very damaging. ' 

That gets us to something I call "justification." If someone has a good feel- 
ing about a particular decision, or they have a decision that they hope is 
going to be made, then they will go looking for data to justify that parti cii-- 
lar decision. In doing so, they are probably going to bump into some 
ihforfetibn'that is not xompletely in support of their decision, ^d they will 
probably change their decision a little Bit. They will bump into a lot «f ^ 
information that says they are all wet. They will have to be changing, 
improving, proving— but I think their finding the, justification for a decision 
is really going to help them gain courage and not make them feel guilty about 
using information to justify a decision they have^made. 

Another way you can make sur% that justification turns into a plus and not a 
crutch is by asking yourself and ypur staff to list the pro'.s and con's of the 
recommendation. If they recommend a particular decision, ask them why it 
shouldn't be done that way. Initially, you will probaWy^shOck people, but 
they.leam rather quickly that you are making sure all the information has 
been considered. We are back to justification again, but we've added a new 
dimension to ft, in that w^ are building in the requirement that they look for 
information that woul<M»*st9 a negative impact on the decision as well as those 
things that support (we decision. 

I have been talking in general terms. Now I would like to talk more specifi- 
cally of the data itself. One time I worked on a report which showed the / 
progress being macfe in a particular school in Louisville as a result of a 
number of innovations. The report referred to the vandalism rate in the 
school. During the middle of the year we began to get very excited al^ut our 
data because we sawJ&B were having ^ tremendous drop in the vandalism' rate, 
and we thought that W^s something extremely positive. We started pattifig 
ourselves orl the bacr a little bit. In several more months the rate went down 
even radre, and then we were convinced that we had something good going unti-l 
we talked with the maintenance man. ' We had determined vandalism by glass 
break^age until we discovered that the maintenance people were puvig in 
ple)^iglass that wouldn't breajt. A drop in glass breakage didn't mean a drop 
in vandalism rates k It just changed the Way vandals operated a little bit. 
One of the dangers in dealing with data lies in not .taking a second look. 
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Before we get too excited about data or start using'data for making decisions, 
we owe It to ourselves to go back and take a good hard look at the data. That 
^! ^J^^y ■•"'PO'^tant step- in utilization of data. You have to sit down and 
Identify both the strengths and weakTiesses of the data to determine the limi- 
tations of U--to know just how far you can go with^if. For example. National 
Assessment data cannot-be .us^_.for cause and effect informatiorr. That's a 

°^ ^'^^'"'^ °^ misuses of data. I think the current misuse ' 

of the ACT and SAT scores by some people is a good example. These two tests 
are designed to depict the success of college f res hinen and they do a very good 
job of predicting the success of. freshmen, but people try to take those tests 
and translate them to see how well we're doing other things in education. It 
IS foolhardy not to consider the weaknesses 'of data--not weaknesses in terms' 
of what they were designed to do, but weaknesses in terms of what some people 
are trying to use the data for.- 
-J 

Another mistake we make with data is to dismiss it rather rapidly. It would 
be- very easy for someone to make the same comment about achievement data for , 
the_ school system as I just made about the SAt and ACT data: "Let's not use 
It because it, has a flaw in it; it is intended for another purpose." That is 
just as serious an error in data utilization because we have to look at data 
to determine what use we may have for it. So, if someone from your staff ' 
rejects a piece of data that you think they should use in reaching decision, 
then again ask why i.t should be used. Very little data generated has'-absol ute- 
ly no value. 

t * 

It is. also important to find out how we feel about the data. Do we like it 
because it supports our position, and that is reason enough for using it? Do 
we tend to disregard it because it tends to be a threat to us, suggesting some 
things that we would just as soon not hear about? Are we a little bit defen- 
sive about the data? We have to ask ourselves all of those questions~To 
determine if we have some barriers -irt^aking use of data that could be ' 
extremely hel pful— barriers of insec^rity that sometimes occur when we look 
at data that doesn't please us. It is extremely important to identify the 
strength's and weaknesses of the data that you project to use and to have an 
open discussion about the way ^ou and yoyr staff feel about that data. 

When yoh go through that k*ind of process, you .can discover the good things and 
the bad things about the data. - You can ther^' communicate that" information back'- 
to whoever produced the data— because thfe producers of data, like National 
Assessment, State or government programs, assessment or evaluation programs, ' 
are all dependent Sin tiw utilization o'f the data in the decision-making . - 
positions.. Ev^eryone in the job" of producing data should welcome constructive 
critici$m; but i;f you go through a process tof identifying strengths and weak- 
nesses of data aiyi keep these to yours>elf, you are not helping yourself. As ' 
far as Natibnal Assessment is concerned, if you let us know ways ip>wh1ch you 
consider the datasto be weak, then we can respond to those weaknfiSCTs; and if ' 
you will let iis .krjpw ways in^which you consider the data" to be strong, then we 
can be Sure that we don't do any^hirtg to threaten the strength of the data. 
But you have to' communicate to data producers a need to change what they are 
doing, improve wKat they are doing, or forget what they are doing and stop 
doing it. . • * 

Any time we make use of^data we make certain assumptions about whpt the'data 
reveals; It is a viise idea to check out your assumptions with the producers 
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of the data so that you feed back informatiorr and check out your assumptions. - 
Maybe the' producer did not realize that people'were making the assumptions, 
you did or that perhaps there was somethin'g else that should have been con-- 
sidered. ^ , ^ ' 

.As you start developing a, personal contact with the generator of the data you 
use in decision-making, it is extremely important that you become an owner of 
that data. There are a couple of ways to do that. I have already mentioned 
some of them. Identifying the strer;igths and weaknesses of the data requires 
an inquiry into the tlajta which causes you to start feeling more comfortable 
with it and it starts becoming a part of you; you now own it because it 1s 
something tKat you havjB worki/ig for you. Listing the pro's and c)>n's of some 

"position that you are taking is another way. to develop ownership because the 
data are^in some way support ivg of where you are coming from. Identifying 
some ways in which everrWfiat you consider to be weak data can be used, even 
that process develops ownership. Personal contact between the* gerterator of 
the- data and you or your staff adds another dimension to that ownership. 
Profi^ibly the most critical thing is that if you are going to use the message 
that the data generator used in generating data for yourself, you ought to 
not>eplicate it— yq|S)ught to refine and improve upon either the methods of 
collex:ting the datalptry to improve the data base by adding additional 
information from other sources. That proJt)ably goes further to create owner- 
ship in th§ data than anything else. 

Finally, we have to watch out not to overrsimpl ify the use of any data. Many 
times* people expect of us to have either yes-no or ei^ther-or types of re- 
sponses. They don't want to deal with complexities^ and so it is very easy 
for us to look at the data, over-simplify, and try to come up with those yes- 
no's, up-down's, either-or's. We're all probably guilty of this mistake at ^ 
some time; so as a final step in the utilization of information in decision- 
making, we have to inquire whether we have tried to over-simplify or whether 
we have dealt with the complexity of the data in oj^d^r to make the decisions 
that will help us do a better job in education. 

Then, when we are faced with momentous decision-making, we can reflect on the 
decisions that we have already made and the information that was used in 
making those decisions, and we will realize that, yes, indeed, we do make use 
of information in trying to replan new educational programs.' And, by using 
some of the strategies that I have mentioned this morning, hopefully we will . 
create a. desire in the people who will have to m^ike use of the information to 
go out and search fan- good information and then to- make the^best use of it 
they can. \ ^ * 
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SUNWARY, EVALUATION AND IMPLEMENTATION . > 

Gerald Kowitz,- University of Okia'homa ' 



Views of the Evaluative Process - 

One of the central problems facing thfs group is fTIar-rrobody likes to' be eval- 

SnJ^ihln"?? r '""^ '^""^ °^ ^^"^^ 9^eat glee in evaluating others, 

but when it is our turn to get under the microscope, it hMrts;. y . ■ 

We have several problems. That means.we must begin'with some axioms. 'things 
that we accept If we are going to get «omething^one. we have to begin somev 
where with what we believe. And for ouh axioms. I propose the foil owtiJ: - 
arpTJnn Jn ^-n ^"^^j^^ble. If youVe spending somebody else's money; they 
iuSamSJf^ ?h?t h^°^.*? Accountability presumes assessment plus 

JJn^'^?"^"^-^^'? ^° *Provement assumes/control . This 

\Jy Jfyj^PO'^tant axiom because it has as a corollary, a non.syn1me{ric: con- ^ 
trol IS based upon knowledge. Control ■ based upon knowledoe has the non-reversible 
nonrsymmetncal corollary that controls cannot give^us knowledge. This J' 
tremely important for evaluation. If we can report no difference, what we're 
S??}J?ence'"' '''' '^'"^ '"^ difference. We Jan't ^Sere was no 

We've only begurf talking about policy. The history of studies of policy l^-e 
^rv^f. a decade old. Oh. we'll fi^d philosophy oix thenr. .You can probably go 
ni?iv5^ ^n!!'-*^'^'* policy-maker in history. Joseph, who save^ fgypt through 
; ^^'^ 2- Vr^^^r, the story of Jethro who'd been to group dynamics 

■ when he told Moses. "You can't be bothered with all these probljems BrSk them 
up.in groups'of ten. tell th^ to go out and ~djscuss it themselves and then 
S hp%n?H"^;:'''°'V You don't believe me? Look it up in Exodus. That's 
nJ JnSt.J? ^u^l' ^'"^ c'""®- W« taJk about policy as a means 

of control-. We have control in physical sciences. We are marveling at our ■ 

Jm^^S^LM'-' 90t It there. It worked. .When you think of the marvelous . 
J5 uu" ^"^^"^ ^'s that they can't bullc^ a smooth 

railroad crb^ng. Why is it that building ventilatioJ-i^ neyer quite right? 
mLc5 Mn^Hw """'^ difficult than putting a man on the moon? 0? a spider on 
Mars? Nobody answers. We have the idea of controlling people and society We 
have ph1 osophy. traditions, law. the social sciences.' We usually thiik of \- - 
them as less precise than the physical sciences. I 'm not so sur6. * 

^c"tl^^'/! ""^l "OV. getting, into the study of the control of the control. -This 
nnhJ ,nH ''" "^^ ethics or philosophy-'phi os^ 

abSit rnJr^]'n/Ih- ^e^Pr*?^^' are talking - 

?ime wp cpp .Mc*^'-""-';°^^'' °^^P°licy. the policy maker. Af the present 
U ?i\-nnpH nfJ^-' political manipulation which, among us./ , 

nnlU;^^n3 ;r P^''^ ^^^^ ^^^^^^ proposal. ■ Somebody has made a 

SSurSl??^ iK^n'^"^"' ^°"^e^ed. ."Why don't you do it ^ , 

yourself? when we do get in^p this arena of social, political manipulation 

'tr'^tSiranna^JfirnJ^: ^feril la warfare. EvaluatfSn plans Se'?Se°"' - 
todai^ arpaf H5?fll °^ trench, warfare, f would suggest to you that one of 
tha? thp^^!! d1ffe.:ences between an educationa'l reseaccher and an evaluator \% 
that the researcher Is looking for solutions problems. The evalulto^is 




trying .to arr^atFiSL^- solutions thatjwilV^^rk, trsi/ig the problems that have been 
st)lved> '.the soljutions to^ tha prp^ems 

^' We^taLk aBout^evaTuation with two faces. We!re lookfTrg in two directions and 
we*r^ seeipg/two^different*^ things. One, we are trying, to justify the past. 
ThiS' f€ 5dQd;/it'has. advaritfiges. We shoeld. substantiate what we have done.. 
Call it justification— thaf*s all right. But let us remember that we can't 
change th.e'past. The second role of evaluation is predicting for the future. 
^ This wiJI help us with bur plans. / ' 
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• Objective-E)^ern^Evaluatio^ 

There are rougflly three different^! eys of the evaluative aj|0j2fess; ^be first 
' J-'like. to call o^a^ctive-externaT #taluat1on— the optimum, me thing^that we • 
all wanted to d6 far years ^nd'years. Yet many, of 'us, who have thought about, 
it .iti 'the dark of the night whenvwe couldn'X^leep, realfzed.^we cw't get - ^ ^ 
there. ..T+ii^-U scientific 'res^estrch' at its best." Objectivity," external , no ^ 
connections, and evaluative*. It is |J|fedictive. if you.do-thts, then this wi>^ 
happen. One of the bases of this '^valDati^^prcrcess is behavio^al objectives. 
You can *not do objectiverexternaV fevaluatibnN^ithowt behavioral objectives. 
But jfst 'Wcaiise '>ou have, behavioral objectives tioes not mean that you Jiave ^1^^ 
object ive^ej^rnal evaluation. : ' ^ 

For example,"^ast. spring I.vas asked to worlc/g^t t^ Local ^^ssociation of Sunday- 
School , Superintendents^ in developing and evaluating their new curriculum proj- 
ects. '-By the time I arrived, all the Sunday school teachers were whiting 
behavioral objectives. I was to 'evaluate ..them. They put me up on the^^odiunl 
' 'and started handing them to me one a\a time. I was doing fairly^well with the 

* idea that after, so msiny sessions the children would know and could name at 
I^L least the 12 idisrt'pTes of Jesus. Then'l came down to one that this little old* 

pMady ti^pwritten. l\ said, ''Of the children that attend. my Sunday-school class 
80 perceht of the time, 50 percent will.^go 'to heaveh." 

Intetiral t<<^his. idea, remember,, is it's external. This means the evaluator 
desire^ as little involvisment i'tf^e program as possible. Ke does not want to , 
.be ^confused' bv the facts. knows what he is doing. It has the great -advan- . 
tage of permitting you to bfe creative. It is supremely %f ens 1^1 1t'fs 
|^thfenattcal»ly<rinrtifiable, it is totally logical, and it has extremely" jifgh 
general izaWl ity'^ because, there is great; focus on the problem at-^lj^nd. ' People 
Wpkjng inT:his Xrea are j30t irrterested in 100,000 case§, becausre they^krfow 
that any test /statistics 'that they apply will be of some significance i'f you' 
have enough cases. , If you ^^enough cases, you will get a statistltally 
signif jcantjJifference even thoQgh it doesn^tmean anything. 

Sarppling 'cannot be well, controlled. We all know that.^Po>^fics frequently 

lere ^ 




^valuator knows whJpfis 
^f the politicalfRplal ♦ 



interfer^witlf the design. There^ is a great temptatltm^to generate case sUidles, 
' resumng 'In' almost tofal||"os$ of generalization. Thi 
r4|lft, and would d6 righ^t,.but cannot do. right because 
implications. , . . • ' 

< S>Jbjective-ExteV^i%^lu.ation J . J''. V ^' * ' • 

' . ^ ■ ' . . ^ "■ ■ ' ■ f • • . ^ ■ 

Next, -r would. like to'dlscuss' subjectlve-exiernal eva4u.ation.7Sre we are get- 
ting the judgment of' expert's who der1it>erat6 on how things ou^htsjo be:. Now; - 



' , ' • • - . '• 

• ' here -Is the difference. Whel^we were talking about the ext€fna?-pbdect1ve, 
there 1s an assumption that policy will autoniatltally flow .from efca^fmental 
•r"\^,-:u "^eny It. ^f we find .these- facts to be self-?^t, thabs 
It. With subjective-externai eva^tlon, .we are going down to allow a little / 
elbow room. > - ■ • , ,- 

M • '■ ■ t ' 

^ Experts wITl .set up standards.- They Wll? dictate to you ^ome policy They will 
expect you liver by it. The operators in the prograrif do no.t set -the quality 
control |0bj^1ves- they are set by the experts. The program directors ha vie .to 
live with |hem somehow.' This is a problem that is locally known as the fa+lacy 
u ^^l^'^^ can conduct any Plaget experiment, you have to be trained 

. by certified Plaget trainers. Tfiey give you your glasses, and looking through- 

t+iose glasses, you can 'only ^e what they have seen. Here you have a very real 
« •problem.. You are merely saying, yes, you saw.whatyou saw. But that might -riot' 
be Ijight. ^ . , • 

u;^ The eyaluator's role in subjective-external evaluation 'is that of the superior 
expjrt.^e hive-put together the«ffs-t brains we-can find, and we've given con- 
. trolXo, the top level.- If we didn't select the right people, we're in trouble. 

This IS a very dangerous', although useful, weappri.^* The evaluators or experts 
^ can come in and criticize you for 'things you never *ri^ to do. They will total- 
ly over^look tMngs that came out of your" plan tha.fdoTi't fit their >econCept1ons. 
In fact, you will .find that this Is one^of the best n]ethodsiif you want to play 
hatchet persoQ. The problem. Is it lacks generaTiiabfttty. - 

■ • • ' 

' Interrfal-Dev/lQpmental Evaluation ' • » x ^ ' • ^ 

Finally, we had a newcomer to the a.rena, only 'abog.t 40 yeahs'old new, AEbrnal- 
-■ dejJiepmental evaluation. This is. only one component. of the p^ram.^fRie • 
^IJV^?*"'' ^ ^^^^ """^^ perfectly tongruvit*w1th that of. the manager, 
.wthat Is, to Improve the program. We are rfot tal^ng about achieving specific 
objettives-what we are talking .about is not wh« the 'organization can do now 
or what it has done in the ^ast, but what Is .its potential for future- perf or- " 
ml/ "j^^^"^ we expand and increase that^)otentiaT. This requ1re&"that 

TV ev^luafor« must be. InvolvedM'rom conception to termination.. 

• Systems analysis Is.a great tool with this method'. rWeWtalkIng about an 
r * ;^vQ)"tiO"arupepat1on. We can't shut down the school to Improve 'it We " - . 
i^y f. have- to Infli^Dce small , bits and see how they work. We Will take' out tbrowaways 
' ' -and put others back in. B'ut we are going to be working td"ie(ier as a team.r 

. , eyaluator and program director, In this repeating process. The^antages hef-e " 
are. the Involvement" and the motivation, for the people concerned. TTnot motlval ' 
T tlon at least cons^ef you are a part of an Important group doing things. 

The disadvantages are very kit^restlng*. The greatest disadvantage Is that you 
are deallng.wit* hl-ghly. abstract Ideas., You may neyer get down to touch earth 
.- . Yourijehayiorally-stated objectives are of little help. You will be r'estrlcted 
, *)y the policy guidelines of the organization.' You must be consciously politl-^ 
/ , cal, and you had better he good at it. You may become, unfortunately, extremely 
glmtpicky-sach as getting pepple together to I'ole-pJay, wearing masks an'd • ' ■/ 
^ - costumesf or .playing at various kinds of gam©s-.7ln-basket, owt-^basket. --We • 
have here the Delphi Technique that more frequently than not bypasses the best 
■ solutions because nobody wants to tair about them at first; ^o they are voted 
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•• down tsight awa'y^ We. have asseriivjeness- training, which frequency deteriorates 
." into attack traThrrig. Evii.luatioi3 is definitely post-hoc. and. lacks generaliz- ^ ' 
. ability. ' '•■.^ • ^ ' ' * * ' 

. Reactions to RIPP Program / m . ^ - . /• 

In my f(nal reaction' U the .RIfP .program, I would stres^the importance of 
*, meeting together face to face as„ pepple^^Wgrds said' to gi^h other, have much ' 
' more meaning than the vibrations that come over the telephone. It is much 

easier to sayno over the telephone Utianjtis ^facd .td face. It is much easier 
^ toMose.a letter j*.n the in-basket or the waste ba^et than^ it i$ to gi\te it a 

possible answer. It is -very important to see pe^bfir^to know people. 

^ If you are going to talk about your orc|anizat^alfl|!y a region^ you have .got to 
get to know that tegfon. . This means somewhere in -your conference I would strongly 
recommend you schedule a regional familtarizatfon activity. People are going to 
' do it anyway. If you can do it with new friends, particularly from the area, 
-you will ,bfegin to get a regional ident*.f ication sfind consolidation that will do 
^ your organization more gocJd than a very large federal grant without, assessment 
attached. Site visits are extrefneTy important. 

I' would siiggest that we need to watch our Tahguage when we start talking about 
Jiow evaJuatiori is too complex' for thosfe poor .dopes. out there. Don't assume 
f:hat they'iare uneducated. They are unwilling to be .-educated. Maybe they need 
a little nourishment before' they .will throw themsel/es into it. I suggested 
' ' to you three modelV.- Please do not consider them «fs' alternatives. And don t 
' ttiink maybe* we can-string them together as^one--each has ns purpose, its 
advantages, its disach^ntages.' 



EVALUATI.pi^0FOR'EFjpTIN5 EDUCATIONAL RTTURES 

>:. ■ ' ^ ■ , 

' Seminar Evaluation Hofm 



DB'g)GRAPHIC : .. ' . 

. 1. Professlbnal Position (Check One): 

^ l<^^lschool district personnel- 
31 state education' agency i^taff 

. ^ ' 1_ education service center, personnel 

1_ higher education faculty or staff 

1 ^eliool board manber 

^ ' * . i.. 
L_ other: Contractor 



2. IVhat State do you represent?- 
QB.JECTIVES: 



3. Were you able identify probable direction^ of change in education and ^ 
ways in which evaluation, co&ld assist with the change process? ' . 

positively yes; generally -only somwAat not at all" ' > 
^ - • 28 . .14 '3 

^' fSdiJIgs?^^^^ ^^"^^ experiences and ideas on evaluation ac,tivitjes and " - 
positively yes, generally only somevAat ^not at all 

'^24 6 .-0- V . 

'vJl^ir^S^i^tlcS? "^^'y ^° rev^i^w^sing or e^gi^g evaluation 
positively yes, generally only iomewhat " not at all 

6. . Did you gain' ideas and suggestions that could be useful in your situation ^ 
. or responsibility? ^ *Ai.u«n.xon . 

.... « 
positively yes, generally only somewhat not at all 

18 • 20 11 , , - '' -0- 
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III- MAJOR' A^IVITIES : ^ (rate the following activities ;\circle your response activity 

in each of fhe three areas note4belq>r) - » 



KNOIVLEDGE 
I GAINED 



POTENTIAL 



o 

iH 



7^ ^RMC Resejarch 
\ Models 

8. Abt Assoc. 
Inc. Models 



Project. \ 

lOf- States "aiow 25 22 
§ Tell 









(/) 

4 ri 


31" 

m o 


4-> 
1 




a* 


2 P 




• 








28 ^ 


^3 




^ 13 


12- 


.29 


4 < 


Z 


25 


.4 


3 


• 10 



f PElCATION ' V 



D CO 

4^ Q 'H 



4-> 

g 



ct g 
a, o o 



-0- 



11- . Utilization 
of Data to 
* make decis. 
. ECS. , 



U 15 



13. 20 
27 » '6 
20 23 2 

'l4 15"^ 7 



5 
8 

2 

■0- 



4 
6 

4 

4" 

7. 



12. Systems ^ 

Development 11 23 
/Corporation 



12 



24 



13 



'13. Note any conments you may have qcfncemjiig the prestentation; 

2 



a* »C - . 

Quiet Technical 

Should li^ve been questioned further 
Not well, presente<i- 

\fell prepared § prpfessionajly" presented 




No CooBient 



30 



Not well presented ' , 

Poorly plamed project of research 
Well prepared arid prresOTted 
Waste of tijoe 
Too much rai^Xisqt 
Cpmnenf « . 

C^if> Eval. Project 

Well prepared ana presented ' 

Missing in tone 

Good practical approach ' 

Geod/Faip 

No COBBUBStt 



6 

T 



s 

T" 



-n- 



'BRESENTATION 



. o 



o 

o 

§ 



» 

7 34- -3 
-0- 53 11 



3 
5 



5 
5 

S 

4 



35 
40 



30 2 12 
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. States Show apa T^ll ' . 
W^ll prepared and presented 
Too -man)^ presentations atone tiine 
Excellent opportunity to cpmpare and share 
Very- Poor/ Fair ^ • , 

Good 0 

•No Cdmment . . . * 

Utilization of Data 

Did not get into depth of problem 

V/ell prepared and- presented 

Fairly informative \ 

Worthwhile - , 

Fair , , * ^ • 

No Comment 



f. Other "Events v *. ' 

Interesting Conference - Well Chrganized 
Wrap-up Evaluation - O.K. 
Speaker was Syperb " •'^ 
U.S.O..E'. Presentation - valuable^id'.useful 
^ Comment • ~ 



8 



- 3 

-'25 



3 

8^ 



3 

T 



28 




22 
1!F 



IV. •0\'ERUL RATING: - . ' - / * • 

14. ^ In general, hbw applicable were seminar. activities to your local sitiiation? 
(circle one) ^ . ' ' ' 



^ highly 
applicable 

^ 11 



generally 
, applicable 

33 



15. 'Coij^^enc^ organization and in^)lein^tation was\ 
supdrb * ' .goocf so^so • ^ 

.15 • ro ' 1 

The conference merai of activities and events was 



16. 



steak 
§ potatoes 

13 



steak potatoes 
20 S 



, hers 
d^oeuvfes 

3 



applicable, 
i- . 



terrilsile 
-0- 

^ 

beans 



no 
comr^nt' 

4 ; 



no. conf^enV , 



-0- 



5s 



othgif;. no- 

tl.*. ^ 8 

' Jim Bean 
Hamburger J5 
i Potatoes' ^ 



V. FOR THE'FimjRE: 



• 17. What should the conference planners have included on the agenda that tKfey^did not 
• include? ' ; ^ 

^to^e direction- ^or policy makers 3 

0 More ''Ideal school district" representation """^ . . , 

More "local school district" pre5entati<)ns 3^ « 

More panel discussions ' Jf" 
r-n!^r- . emphai3is on what actually is going on at 

tl\l^L . , state and local Ipvels P{ ^ 



Good' job • , , 

Good effort ' * . ^ 

UjS.O.E. nreactions 
More interaction with gFOups^ 
^to^e time on the Calif / Evaluation 
^Training package 
No Conment 



17. Anything' Else? 



^2 



Field Te^t Future Forms 


2 


Group* exerc^se^ well liked 


3 


PaneX most. informative, and useful 




More real discuss ion^ * 


r 


Good conference 




To intense - ijiore varied- ' . • 




To much* time 


1 


Final suiftmary overview' - excellent 


• 29" 


No Comnient - ^ 
'4 • • 


0 
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g^f^ENTS t>N KEYNOTE' SPEAKER: ' ' / ^ ^ 

s , ♦ 

1. Excellent, set 'right, tone ^ for the entire conference^ 

2. Ver)! interesting. • * ^ *\ • 

3. '^Heavenly ifiinded" • only one point retained- ijnportance of process 
evaluation couched .in lofty terms . ♦ . 

4. ' Very goSd-and controversial. , . ' ^ . 

5. Very ^interesting, agreed with biases/donvictions so, therefore, di^i. 
not introduce many new ideas but did serve to support and facilitate 
my own thinking^ on many issues. • V 

6. Enjoytsd the speajcpr very much. Did not learn much that I didn\^ already 
^know but he did serve to refresh some philosophical ideas that Set the 
scene for the conference. * ^ • ^ 

^ 7. Somev^t removed from the real world of where the ihterfact between kids 
and programs occurs. ' \ % 

* ^ 8. He was a good speaker. Tried to get j^cross a. balance between objective 
and subjective assessment;. Perhaps a little too general*. 
9. Very perceptive abou^ future ne6ds regarding education. 

10. Afraid I didn't get much out of it - lots of platitjudes etc. • 

11. ^cellent choice and did an- excellent job.. He was ''down to earth'^ 
\12. Stimulating, controversial, got us thinking and new ideas. 

15. A small, small- peek tjhrpugb a tilhy crack in .the evaluation door. Humor, 

. . A-*-* ^ • * ^ - 

l4. Dyiiaitilc; set the ri^t tone.*^ Easy, to follow and hurnprous* 
15: Average. ' ; * . ^ ,^ 

16. Very good, brought good information. 

17. ' I appreciated the keynote address prii^rily because of its "scholarly" 

tone, which probably isn*t appropriate for a summary . This is the first 
session, confereitoe, meeting, etc; I ha>ffi- attended which had a s^gXe 
^scholarj[y luidress;-it, was great. I was Mginning to think (after Tenths 
• with the State Department of Education^ that intellectual ism was dea^ ^ 
among "professional educators'i. . , ^ 
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VII. SUNMARY SPEAKER: 



1. 
2. 

3/ 

4. ' 



5. 
6. 

• 

7. 
8. 



9. 

10. 

11. 
12. 

13. 
14. 
15. 



• "Earthly- Good''. .Entertaining, informative, and excellent deHriefing 
which wove the various experiences of the sessipn as threads in a tapestry, 
After hearing the Summary Speaker, I had a*irucn better idea of what the 
Keynote Speaker said - I understood the sunmary version best. 
Very ponprehensive and perceptive covferage of the topics/issues raisjed 
during the conference; very stihiulating presentation; good speaker; share 
important insights into the evaluation process. ^ 
1 enjoyed his practical approach to evaluation. ,His humor added to the 
presentation, I agreed with his ideas that some speakers wjere patronizing 
^ tJ^at the program should have had more time for sharing projects and 
ideas excellent. 

Closer to the real world of the classro(OT. . 
Entertaining but took many easy craclcs that were unjustified. Didn't ^ / 
really have anything to say.' ^* 
Very good stSfnmary of conference. ^ . ' . > ) 

One of the finest wrap-ups I have ever heard. Gla4 we can laugh a\ our- 
selves and share sOTie serious thoughts too. It's nice to have ^someone , 
give a pretty objective coimientary on the state of the art. 
Very entertaining and ''on target". He did an excellent job: on putting the 
spealjters in perspective. ^ * ' . 

Went beyond summary to conts(ibute much .itself. Qood, on target - could 
have been just a little briefer.' _ ' , 

Insightful. Suggestions for meeting formats were those I can support. 
Nkjre exciting and well planned summary than I have ever heard. It was 
exciting, humorous, well planned and to the point . , 



Very good - excellent. 
Enjoyable and informative 
Superb - excellent. 



very good. 



VIII. BOTH SPEAKERS: 



1. Both the Keynote and Sunmary Speakers were excelftnt. I p^rticulafly liked 
the Summary Speaker as he injected many fu%e stories and information. 

2. Both are "ben of kqpressive intelligence and provpcotive imagination - 
excellent. ( ' . ^ 

3. Best parts of conference. Demonstrated more perspectrle on evaluation than 
others combined. Matute, professional educators -;^covided>enthusiasm and 

• hope for work. . ^ . \ I ^ ; " . 

,4^^. Both Speakers werfe very ^xki ancT controversial.' 

5. Bftth Speakers offered an appropriate spark - the initial spark to excite;- 
the concluding spark to sustain interest. Please print remarks. Both had 
gr^at points that merit further thought. 

6. Both were interesting, very informative - excellent. 
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